Scalability techniques based on content information

ABSTRACT

Apparatus and methods of using content information for encoding multimedia data are described. A method of processing multimedia data includes classifying content of multimedia data, and encoding the multimedia data in a first data group and in a second data group based on the content classification, wherein the first data group comprises a coefficient and the second data group comprises a first differential refinement associated with the first data group coefficient. An apparatus for using content information for encoding multimedia data includes a content classifying module configured to classify content of multimedia data and provide content classification data, and an encoder configured to encode the multimedia data in a first data group and in a second data group based on the content classification, wherein the first data group comprises a coefficient and the second data group comprises a first differential refinement associated with the first data group coefficient.

CLAIM OF PRIORITY UNDER 35 U.S.C. §119

The present application for patent claims priority to (a) ProvisionalPatent Application No. 60/721,416 entitled “A VIDEO TRANSCODER FORREAL-TIME STREAMING AND MOBILE BROADCAST APPLICATIONS” filed Sep. 27,2005, (b) Provisional Patent Application No. 60/789,377 entitled “AVIDEO TRANSCODER FOR REAL-TIME STREAMING AND MOBILE BROADCASTAPPLICATIONS” filed Apr. 4, 2006, (c) Provisional Application No.60/727,643 entitled “METHOD AND APPARATUS FOR SPATIO-TEMPORALDEINTERLACING AIDED BY MOTION COMPENSATION FOR FIELD-BASED VIDEO” filedOct. 17, 2005, (d) Provisional Application No. 60/727,644 entitled“METHOD AND APPARATUS FOR SHOT DETECTION IN VIDEO STREAMING” filed Oct.17, 2005, (e) Provisional Application No. 60/727,640 entitled “A METHODAND APPARATUS FOR USING AN ADAPTIVE GOP STRUCTURE IN VIDEO STREAMING”filed Oct. 17, 2005, (f) Provisional Application No. 60/730,145 entitled“INVERSE TELECINE ALGORITHM BASED ON STATE MACHINE” filed Oct. 24, 2005,and (g) Provisional Application No. 60/789,048 entitled “SPATIO-TEMPORALDEINTERLACING AIDED BY MOTION COMPENSATION FOR FIELD-BASED MULTIMEDIADATA” filed Apr. 3, 2006. All seven of these provisional patentapplications are assigned to the assignee hereof and hereby expresslyincorporated by reference herein.

REFERENCE TO CO-PENDING APPLICATIONS FOR PATENT

The present application for patent is related to U.S. patent applicationSer. No. 11/373,577 entitled “CONTENT CLASSIFICATION FOR MULTIMEDIAPROCESSING” filed on Mar. 10, 2006, assigned to the assignee hereof andhereby expressly incorporated by reference herein.

BACKGROUND

1. Field

The present application is directed to apparatus and methods for videotranscoding of video data for real-time streaming and, moreparticularly, to transcoding video data for real-time streaming inmobile broadcast application.

2. Background

Efficient video compression is useful in many multimedia applicationssuch as wireless video streaming and video telephony, due to the limitedbandwidth resources and the variability of available bandwidth. Certainvideo coding standards, such as MPEG-4 (ISO/IEC), H.264 (ITU), orsimilar video coding provide high efficiency coding well suited forapplications such as wireless broadcasting. Some multimedia data, forexample, digital television presentations, is generally coded accordingto other standards such as MPEG-2. Accordingly, transcoders are used totranscode or convert multimedia data coded according to one standard(e.g., MPEG-2) to another standard (e.g., H.264) prior to wirelessbroadcasting.

Improvements rate optimized codecs could offer advantages in errorresiliency, error recovery, and scalability. Moreover, use ofinformation determined from the multimedia data itself could also offeradditional improvements for encoding, including error resiliency, errorrecovery, and scalability. Accordingly, a need exists for a transcoderthat provide highly efficient processing and compression of multimediadata that uses information determined from the multimedia data itself,is scalable, and is error resilient for use in many multimedia dataapplications including mobile broadcasting of streaming multimediainformation.

SUMMARY

Each of the inventive content based transcoding apparatuses and methodsdescribed and illustrated has several aspects, no single one of which issolely responsible for its desirable attributes. Without limiting thescope of this disclosure, its more prominent features will now bediscussed briefly. After considering this discussion, and particularlyafter reading the section entitled “Detailed Description” one willunderstand how the features of this content driven transcoding providesimprovements for multimedia data processing apparatuses and methods.

Inventive aspects described herein relate to using content informationfor various methods of encoding multimedia data and in various modulesor components of an encoder, for example, an encoder used in atranscoder. A transcoder can orchestrate transcoding multimedia datausing content information. The content information can be received fromanother source, for example, metadata that is received with the video.The transcoder can be configured to generate content information througha variety of different processing operation. In some aspects, thetranscoder generates a content classification of the multimedia data,which is then used in one or more encoding processes. In some aspects, acontent driven transcoder can determine spatial and temporal contentinformation of the multimedia data, and use the content information forcontent-aware uniform quality encoding across channels, and contentclassification based compression/bit allocation.

In some aspects, content information (e.g., metadata, content metricsand/or a content classification) of multimedia data is obtained orcalculated, and then provided to components of the transcoder for use inprocessing the multimedia data for encoding. For example, a preprocessorcan use certain content information for scene change detection,performing inverse telecine (“IVTC”), de-interlacing, motioncompensation and noise suppression (e.g., 2D wavelet transform) andspatio-temporal noise reduction, e.g., artifact removal, de-ringing,de-blocking, and/or de-noising. In some aspects, a preprocessor can alsouse the content information for spatial resolution down-sampling, e.g.,determining appropriate “safe” and “action handling” areas whendown-sampling from standard definition (SD) to Quarter Video GraphicsArray (QVGA).

In some aspects, an encoder includes a content classification modulethat is configured to calculate content information. The encoder can usecontent classification for bit rate control (e.g., bit allocation) indetermining quantization parameters (QP) for each MB, for motionestimation, for example, performing color motion estimation (ME),performing motion vector (MV) prediction, scalability in providing abase layer and an enhancement layer, and for error resilience by using acontent classification to affect prediction hierarchy and errorresiliency schemes including, e.g., adaptive intra refresh, boundaryalignment processes, and providing redundant I-frame data in anenhancement layer. In some aspects, the transcoder uses the contentclassification in coordination with a data multiplexer for maintainingoptimal multimedia data quality across channels. In some aspects, theencoder can use the content classification information for forcingI-frames to periodically appear in the encoded data to allow fastchannel switching. Such implementations can also make use of I-blocksthat may be required in the encoded data for error resilience, such thatrandom access switching and error resilience (based on, e.g., contentclassification) can be effectively combined through prediction hierarchyto improve coding efficiency while increasing robustness to errors.

In one aspect a method of processing multimedia data comprisesclassifying content of multimedia data, and encoding the multimedia datain a first data group and in a second data group based on the contentclassification, wherein the first data group comprises a coefficient andthe second data group comprises a first differential refinementassociated with the first data group coefficient. The encoding caninclude determining a bit rate based on the content classification ofthe multimedia data, and encoding the multimedia data based on the bitrate. Classifying content can comprise determining complexity of themultimedia data, and wherein the selected multimedia data is encodedbased on the complexity of the multimedia data. The complexity cancomprise temporal complexity or spatial complexity, or temporalcomplexity and spatial complexity. The encoding can include encoding themultimedia data so as to allow decoding of only the first data group orthe first data group and the second data group into a single combineddata group. The first differential refinement can indicate a differencebetween a selected video frame and frame data resulting from decodingthe first data group. The first data group can be a base layer and thesecond data group can be an enhancement layer. In addition, the methodcan include selecting the coefficient from one of an original base layerresidual error coefficient or an original enhancement layer residualerror coefficient, and calculating the first differential refinementbased on the coefficient and the original enhancement layer residualerror coefficient. Encoding can further comprise encoding macroblockheader information and motion vector information in the first datagroup. Encoding can further comprise quantizing the first data group ata first step size, and quantizing the second data group at a second stepsize, wherein the first step size and second step size are related by ascale factor. Encoding can further include determining a firstquantization parameter having a first quantization step size for use inencoding the first data group, and determining a second quantizationparameter having a second quantization step size for use in encoding thesecond data group, wherein the first and second quantization parametersare determined based on content information of a selected frame data,and wherein said first quantization step size is coarser than saidsecond quantization step size. In another aspect, encoding includesencoding the first data group using I-frames, and P-frames or anycombination thereof and encoding the second data group using I-frames,P-frames, and B-frames or any combination thereof.

In another aspect, an apparatus for encoding multimedia data includesmeans for classifying content of multimedia data, means for encoding themultimedia data in a first data group and in a second data group basedon the content classification, wherein the first data group comprises acoefficient and the second data group comprises a first differentialrefinement associated with the first data group coefficient. Theencoding means can comprise means for determining a bit rate based onthe content classification of the multimedia data, and encoding themultimedia data based on the bit rate. The classifying content means caninclude means for determining complexity of the multimedia data, andwherein the selected multimedia data is encoded based on the complexityof the multimedia data, the complexity comprising temporal complexity orspatial complexity, or temporal complexity and spatial complexity. Theencoding means can comprise means to allow decoding of only the firstdata group or the first data group and the second data group into asingle combined data group.

In another aspect, an apparatus includes a content classifying moduleconfigured to classify content of multimedia data and provide contentclassification data, and an encoder configured to encode the multimediadata in a first data group and in a second data group based on thecontent classification, wherein the first data group comprises acoefficient and the second data group comprises a first differentialrefinement associated with the first data group coefficient. The encodercan include a bit rate component configured to determine a bitallocation based on the content classification, and wherein the encodingcomponent is further configured to encode the selected multimedia datausing the bit allocation.

In another aspect, a machine-readable medium comprises instructions thatupon execution cause a machine to classify content of multimedia data,and encode the multimedia data in a first data group and in a seconddata group based on the content classification, wherein the first datagroup comprises a coefficient and the second data group comprises afirst differential refinement associated with the first data groupcoefficient.

Another aspect a processor being configured to classify content ofmultimedia data, and encode the multimedia data in a first data groupand in a second data group based on the content classification, whereinthe first data group comprises a coefficient and the second data groupcomprises a first differential refinement associated with the first datagroup coefficient.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a block diagram of a media broadcast system including atranscoder for transcoding between different video formats.

FIG. 1B is a block diagram of an encoder configured to encode multimediadata and provide an encoded first data group and an encoded second datagroup.

FIG. 1C is a block diagram of a processor configured to encodemultimedia data.

FIG. 2 is a block diagram of an example of the transcoder of the systemof FIG. 1.

FIG. 3 is a flow diagram illustrating the operation of a parser usedwithin the transcoder of FIG. 2.

FIG. 4 is a flow diagram illustrating the operation of a decoder usedwithin the transcoder of FIG. 2.

FIG. 5 is a system timing diagram illustrating a sequence of operationsperformed by the transcoder of FIG. 2.

FIG. 6 is a flow diagram illustrating a sequence of operations andfunctions of a preprocessor that may be used in the transcoder of FIG.2.

FIG. 7 is a block diagram of an exemplary 2-pass encoder that may beused in the transcoder of FIG. 2.

FIG. 8 illustrates an example of a classification chart that illustratesone aspect of how associating texture and motion values with contentclassification.

FIG. 9 is a flow diagram illustrating an exemplary operation for contentclassification, such as for use in the encoder of FIG. 7.

FIG. 10 is a flow diagram illustrating the operation of a rate control,such as for use with the encoder of FIG. 7.

FIG. 11 is a flow diagram illustrating the operation of an exemplarymotion estimator, such as for use with the encoder of FIG. 7.

FIG. 12 is a flow diagram illustrating the operation of an exemplarymode decision encoder function, such as for use with the encoder of FIG.7

FIG. 13 is a flow diagram illustrating an exemplary operation effectingscalability for use in the encoder of FIG. 7.

FIG. 14 is a flow diagram illustrating an exemplary operation effectingrate-distortion data flow as occurs in the encoder FIG. 7, for example.

FIG. 15 is a graph illustrating the relationship between the encodingcomplexity, allocated bits, and human visual quality.

FIG. 16 is a graph illustrating a non-linear scene detection formula.

FIG. 17A is a flow diagram illustrating processing multimedia data thathas been obtained, received, or is otherwise accessible.

FIG. 17B is a block diagram of a multimedia encoding system.

FIG. 18 is a diagram illustrating a deinterlacing process using motionestimation/compensation.

FIG. 19 is a block diagram of a multimedia communication system.

FIG. 20 is a diagram illustrating the organization of a video bitstreamin an enhancement layer and a base layer.

FIG. 21 is a diagram illustrating the alignment of slices to video frameboundaries.

FIG. 22 is a block diagram illustrating prediction hierarchy.

FIG. 23 is a process flow diagram illustrating a method of encodingmultimedia data based on the content information.

FIG. 24 is a process flow diagram illustrating a method of encodingmultimedia data so as to align data boundaries based on contentinformation level.

FIG. 25 is a graphic illustrating a safe action area and a safe titlearea of a frame of data.

FIG. 26 is a graphic illustrating a safe action area of a frame of data.

FIG. 27 is a process flow diagram illustrating a process of encodingmultimedia data using adaptive intra refresh based on multimedia contentinformation.

FIG. 28 is a process flow diagram illustrating a process of encodingmultimedia data using redundant I frames based on multimedia contentinformation.

FIG. 29 illustrates motion compensation vectors between a current frameand a previous frame MV_(p) and a current frame and a next frame MV^(N).

FIG. 30 is a process flow diagram illustrating shot detection.

FIG. 31 is a process flow diagram illustrating encoding base andenhancement layers.

FIG. 32 is a schematic illustrating encoding a macroblock.

FIG. 33 is a schematic illustrating modules for encoding a base layerand an enhancement layer.

FIG. 34 shows an example of a base layer and enhancement layercoefficient selector process.

FIG. 35 shows another example of a base layer and enhancement layercoefficient selector process.

FIG. 36 shows another example of a base layer and enhancement layercoefficient selector process.

FIG. 37 is a process flow diagram illustrating encoding multimedia databased on content information.

FIG. 38 is a diagram illustrating possible system decisions in aninverse telecine process.

FIG. 39 illustrates boundaries in a macroblock to be filtered by adeblocking process.

FIG. 40 is a diagram illustrating a spatio-temporal deinterlacingprocess.

FIG. 41 illustrates an example of 1-D poly-phase resampling.

FIG. 42 is a flow diagram illustrating an example of adaptive GOPstructure in video streaming.

It is noted that, where appropriate, like numerals refer to like partsthroughout the several views of the drawings.

DETAILED DESCRIPTION

The following detailed description is directed to certain aspectsdiscussed in this disclosure. However, the invention can be embodied ina multitude of different ways. Reference in this specification to “oneaspect” or “an aspect” means that a particular feature, structure, orcharacteristic described in connection with the aspect is included in atleast one aspect. The appearances of the phrase “in one aspect,”“according to one aspect,” or “in some aspects” in various places in thespecification are not necessarily all referring to the same aspect, norare separate or alternative aspects mutually exclusive of other aspects.Moreover, various features are described which may be exhibited by someaspects and not by others. Similarly, various requirements are describedwhich may be requirements for some aspects but not other aspects.

The following description includes details to provide a thoroughunderstanding of the examples. However, it is understood by one ofordinary skill in the art that the examples may be practiced even ifevery detail of a process or device in an example or aspect is notdescribed or illustrated herein. For example, electrical components maybe shown in block diagrams that do not illustrate every electricalconnection or every electrical element of the component in order not toobscure the examples in unnecessary detail. In other instances, suchcomponents, other structures and techniques may be shown in detail tofurther explain the examples.

The present disclosure relates to controlling encoding and transcodingapparatus and methods using content information of the multimedia databeing encoded. “Content information” or “content” (of the multimediadata) are broad terms meaning information relating to the content ofmultimedia data and can include, for example, metadata, metricscalculated from the multimedia data and content related informationassociated with one or more metrics, for example a contentclassification. Content information can be provided to an encoder ordetermined by an encoder, depending on the particular application. Thecontent information can be used for many aspects of multimedia dataencoding, including scene change detection, temporal processing,spatio-temporal noise reduction, down-sampling, determining bit ratesfor quantization, scalability, error resilience, maintaining optimalmultimedia quality across broadcast channels, and fast channelswitching. Using one ore more of these aspects, a transcoder canorchestrate processing multimedia data and produce content-relatedencoded multi-media data. Descriptions and figures herein that describetranscoding aspects can also be applicable to encoding aspects anddecoding aspects.

The transcoder apparatus and methods relates to transcoding from oneformat to another, and is specifically described herein as relating totranscoding MPEG-2 video to enhanced, scalable H.264 format fortransmission over wireless channels to mobile devices, illustrative ofsome aspects. However, the description of transcoding MPEG-2 video toH.264 format is not intended as limiting the scope of the invention, butis merely exemplary of some aspects of the invention. The disclosedapparatus and methods provide a highly efficient architecture thatsupports error resilient encoding with random access and layeringcapabilities, and can be applicable to transcoding and/or encoding videoformats other than MPEG-2 and H.264 as well.

“Multimedia data” or simply “multimedia” as used herein, is a broad termthat includes video data (which can include audio data), audio data, orboth video data and audio data. “Video data” or “video” as used hereinas a broad term referring to frame-based or field-based data, whichincludes one or more images or related sequences of images, containingtext, image information and/or audio data, and can be used to refer tomultimedia data (e.g., the terms can be used interchangeably) unlessotherwise specified.

Described below are examples of various components of a transcoder andexamples of processes that can use content information for encodingmultimedia data.

FIG. 1A is block diagram illustrating a data flow of some aspects of amultimedia data broadcast system 100. In system 100, a multimedia dataprovider 106 communicates encoded multimedia data 104 to a transcoder200. The encoded multimedia data 104 is received by the transcoder 200,which processes the multimedia data 104 into raw multimedia data inblock 110. The processing in block 110 decodes and parses the encodedmultimedia data 104, and further processes the multimedia data toprepare it for encoding into another format. The decoded multimedia datais provided to block 112 where the multimedia data is encoded to apredetermined multimedia format or standard. Once the multimedia datahas been encoded, at block 114 it is prepared for transmission, via forexample, a wireless broadcast system (e.g., a cellular phone broadcastnetwork, or via another communication network). In some aspects, thereceived multimedia data 104 has been encoded according to the MPEG-2standard. After the transcoded multimedia data 104 has been decoded, thetranscoder 200 encodes the multimedia data to an H.264 standard.

FIG. 1B is a block diagram of an transcoder 130 that can be configuredto perform the processing in blocks 110 and 112 of FIG. 1A. Thetranscoder 130 can be configured to receive multimedia data, decode andparse the multimedia data into packetized elementary streams (e.g.,subtitles, audio, metadata, “raw” video, CC data, and presentation timestamps), encode the into a desired format, and provide encoded data forfurther processing or transmission. The transcoder 130 can be configuredto provide encoded data in an two or more data groups, for example, anencoded first data group and an encoded second data group, which isreferred to as layered encoding. In some examples of aspects, thevarious data groups (or layers) in a layered encoding scheme can beencoded at different levels of quality, and formatted such that dataencoded in a first data group is of a lower quality (e.g., provides alower visual quality level when displayed) than data encoded in a seconddata group.

FIG. 1C is a block diagram of a processor 140 that can be configured totranscode multimedia data, and can be configured to perform a portion orall of the processing depicted in blocks 110 and 112 of FIG. 1A.Processor 140 can include modules 124 a . . . n perform one or more ofthe transcoding processes described herein, including decoding, parsing,preprocessing, and encoding, and use content information for processing.Processor 140 also includes internal memory 122 and can be configured tocommunicate with external memory 120, either directly or indirectlythrough another device. The processor 140 also includes a communicationsmodule 126 configured to communicate with one or more devices externalto the processor 140, including to receive multimedia data and toprovide encoded data, such as data encoded in a first data group anddata encoded in a second data group. In some examples of aspects, thevarious data groups (or layers) in a layered encoding scheme can beencoded at different levels of quality, and formatted such that dataencoded in a first data group is of a lower quality (e.g., provides alower visual quality level when displayed) than data encoded in a seconddata group.

The transcoder 130 or the preprocessor 140 (configured for transcoding)components thereof, and processes contained therein, can be implementedby hardware, software, firmware, middleware, microcode, or anycombination thereof. For example, a parser, decoder, preprocessor, orencoder may be standalone components, incorporated as hardware,firmware, middleware in a component of another device, or be implementedin microcode or software that is executed on a processor, or acombination thereof. When implemented in software, firmware, middlewareor microcode, the program code or code segments that perform the motioncompensation, shot classifying and encoding processes may be stored in amachine readable medium such as a storage medium. A code segment mayrepresent a procedure, a function, a subprogram, a program, a routine, asubroutine, a module, a software package, a class, or any combination ofinstructions, data structures, or program statements. A code segment maybe coupled to another code segment or a hardware circuit by passingand/or receiving information, data, arguments, parameters, or memorycontents.

Illustrative Example of a Transcoder Architecture

FIG. 2 illustrates a block diagram of an example of a transcoder thatmay be used for the transcoder 200 illustrated in the multimediabroadcast system 100 of FIG. 1. The transcoder 200 comprises aparser/decoder 202, a preprocessor 226, an encoder 228, and asynchronizing layer 240, further described below. The transcoder 200 isconfigured to use content information of the multimedia data 104 for oneor more aspects of the transcoding process, as is described herein.Content information can be obtained from a source external to thetranscoder 200, through multimedia metadata, or calculated by thetranscoder, for example, by the preprocessor 226 or the encoder 228. Thecomponents shown in FIG. 2 are illustrative of an components that can beincluded in a transcoder that uses content information for one or moretranscoding processes. In a particular implementation, one or more ofthe components of the transcoder 200 may be excluded or additionalcomponents may be included. Additionally, portions of the transcoder andtranscoding processes are described so as to allow someone of skill inthe art to practice the invention even if every detail of a process or adevice may not be described herein.

FIG. 5 illustrates a timing diagram as a graphical illustration oftemporal relationships of the operation of various components and/orprocesses of the transcoder 200. As shown in FIG. 5, encoded streamingvideo 104 (encoded multimedia data), such as MPEG-2 video, is firstreceived at an arbitrary time zero (0) by the parser 205 (FIG. 2). Nextthe video stream is parsed 501, demultiplexed 502 and decoded 503, suchas by parser 205 in combination with decoder 214. As illustrated, theseprocesses can occur in parallel, with slight timing offset, in order toprovide stream output of processing data to the preprocessor 226 (FIG.2). At a time T₁ once the preprocessor 226 has received enough data fromdecoder 214 to begin outputting processing results, the remainingprocessing steps become sequential in nature, with first pass encoding505, second pass encoding 506, and re-encoding 507 occurring in sequenceafter preprocessing until completion of re-encoding at a time T_(f).

The transcoder 200 described herein can be configured to transcode avariety of multimedia data, and many of the processes apply to whatevertype of multimedia data is transcoded. Although some of the examplesprovided herein relate particularly to transcoding MPEG-2 data to H.264data, these examples are not meant to limit the disclosure to such data.Encoding aspects described below can be applied to transcoding anysuitable multimedia data standard to another suitable multimedia datastandard.

Parser/Decoder

Referring again to FIG. 2, the parser/decoder 202 receives multimediadata 104. Parser/decoder 202 includes a transport stream parser(“parser”) 205 that receives the multimedia data 104 and parses the datainto a video elementary stream (ES) 206, an audio ES 208, presentationtime stamps (PTS) 210 and other data such as subtitles 212. An EScarries one type of data (video or audio) from a single video or audioencoder. For example, a video ES comprises the video data for a sequenceof data, including the sequence header and all the subparts of thesequence. A packetized elementary stream, or PES, consists of a singleES which has been made into packets, each typically starting with anadded packet header. A PES stream contains only one type of data fromone source, e.g. from one video or audio encoder. PES packets havevariable length, not corresponding to the fixed packet length oftransport packets, and may be much longer than a transport packet. Whentransport packets are formed from a PES stream, the PES header can beplaced at the beginning of a transport packet payload, immediatelyfollowing the transport packet header. The remaining PES packet contentfills the payloads of successive transport packets until the PES packetis all used. The final transport packet can be filled to a fixed length,e.g., by stuffing with bytes, e.g., bytes=0×FF (all ones).

The parser 205 communicates the video ES 206 to a decoder 214 which ispart of the parser/decoder 202 shown here. In other configurations theparser 205 and the decoder 214 are separate components. The PTS 210 aresent to a transcoder PTS generator 215, which can generate separatepresentation time stamps particular to the transcoder 200 for use inarranging data to be sent from the transcoder 200 to a broadcast system.The transcoder PTS generator 215 can be configured to provide data to async layer 240 of the transcoder 200 to coordinate the synchronizationof the data broadcast.

FIG. 3 illustrates a flow diagram of one example of a process 300 thatthe parser 205 may follow when parsing out the various packetizedelementary streams described above. Process 300 starts at block 302 whenmultimedia data 104 is received from a content provider 106 (FIG. 1).Process 300 proceeds to block 304 where initialization of the parser 205is performed. Initialization may be triggered by an independentlygenerated acquisition command 306. For example, a process that isindependent from the parser 205 and is based on an externally receivedTV-schedule and channel lineup information may generate the acquisitioncommand 306. Additionally, real-time transport stream (TS) bufferdescriptors 308 may be input to assist in both initialization and formain processing.

As illustrated in block 304, initialization can include acquiring acommand syntax verification, performing a first pass PSI/PSIP/SI(program specific information/program and system informationprotocol/system information) processing, performing processingspecifically related to either the acquisition command or thePSI/PSIP/SI consistency verification, allocating a PES buffers for eachPES, and setting timing (e.g., for alignment with desired acquisitionstart instant). The PES buffers hold the parsed ES data and communicateeach parsed ES data to a corresponding audio decoder 216, text encoder220, decoder 214, or transcoder PTS generator 215.

After initialization, process 300 proceeds to block 310 for mainprocessing of the received multimedia data 104. Processing in block 310can include target packet identifier (PID) filtering, continuousPSI/PSIP/SI monitoring and processing, and a timing process (e.g., forachieving a desired acquisition duration) so that the incomingmultimedia data is passed into the appropriate PES buffers. As a resultof processing the multimedia data in block 310, a program descriptor andindication of the PES buffer ‘read’ are generated, which will interfacewith the decoder 214 (FIG. 2) as described hereinbelow.

After block 310, the process 300 proceeds to block 314, wheretermination of the parsing operations occur, including generating atimer interrupt and freeing of PES buffers consequent to theirconsumption. It is noted that PES buffers will exist for all relevantelementary streams of the program cited in its descriptor such as audio,video, and subtitle streams.

Referring again to FIG. 2, the parser 205 sends the audio ES 208 to anaudio decoder 216 for corresponding to the transcoder implementation andprovides the encoded text 250 to the synch layer 240 and decoding of theaudio information. The subtitle information 212 is delivered to a textencoder 220. Closed captioning (CC) data 218 from a decoder 214 is alsoprovided to the text encoder 220, which encodes the subtitle information212 and the CC data 218 in a format effected by the transcoder 200.

The parser/decoder 202 also includes the decoder 214, which receives thevideo ES 206. The decoder 214 can generate metadata associated withvideo data, decodes the encoded video packetized elementary stream intoraw video 224 (for example, in standard definition format), andprocesses the video closed captioned data in the video ES stream.

FIG. 4 shows a flow diagram illustrating one example of a decodingprocess 400 that can be performed by the decoder 214. Process 400 startswith input of a video elementary stream data 206 at block 402. Theprocess 400 proceeds to block 404 where the decoder is initialized.Initialization may include a number of tasks, including detection of avideo sequence header (VSH), performing first pass VSH, video sequence(VS), and VS Display Extension processing (including video format, colorprimaries, and matrix coefficients), and allocating data buffers torespectively buffer the decoded picture, associated metadata and closedcaption (CC) data. Additionally, the video PES buffer ‘read’ information406 provided by the parser 205 is input (e.g., which can be generated byprocess 300 in block 310 of FIG. 3).

After initialization at block 404, the process 400 proceeds to block 408where the main processing of the video ES is performed by the decoder214. Main processing includes polling the video PES buffer ‘read’information or “interface” for new data availability, decoding the videoES, reconstructing and storing pixel data at picture boundariessynchronizing, video & a/v generating metadata and storing at pictureboundaries, and CC data storing at picture boundaries. The results block410, of the main processing 408 includes generation of sequencedescriptors, decoded picture buffer descriptors, metadata bufferdescriptors, and CC data buffer descriptors.

After the main processing 408, process 400 proceeds to block 412 whereit performs a termination process. The termination process can includedetermining termination conditions, including no new data occurring fora particular duration above a predetermined threshold, detection of asequence end code, and/or detection of an explicit termination signal.The termination process can further include freeing decoded picture,associated metadata, and CC data buffers consequent to their consumptionby a preprocessor to be described below. Process 400 ends block 414,where it can enter a state of waiting for video ES to be received asinput.

Preprocessor

FIG. 2, and in more detail FIG. 6, illustrate a sample aspect of apreprocessor 226 that can use content information for one or morepreprocessing operations. Preprocessor 226 receives metadata 222 anddecoded “raw” video data 224 from the parser/decoder 202. Thepreprocessor 226 is configured to perform certain types of processing onthe video data 224 and the metadata 222 and provide processed multimedia(e.g., base layer reference frames, enhancement layer reference frames,bandwidth information, content information) and video to the encoder228. Such preprocessing of multimedia data can improve the visualclarity, anti-aliasing, and compression efficiency of the data.Generally, the preprocessor 226 receives video sequences provided by thedecoder 214 in the parser decoder 202 and converts the video sequencesinto progressive video sequences for further processing (e.g., encoding)by the encoder 228. In some aspects, the preprocessor 226 can beconfigured for numerous operations, including inverse telecine,deinterlacing, filtering (e.g., artifact removal, de-ringing,de-blocking, and de-noising), resizing (e.g., spatial resolutiondown-sampling from standard definition to Quarter Video Graphics Array(QVGA)), and GOP structure generation (e.g., calculating complexity mapgeneration, scene change detection, and fade/flash detection).

The preprocessor 226 can use metadata from the decoder to affect one ormore of the preprocessing operations. Metadata can include informationrelating to, describing, or classifying the content of the multimediadata (“content information”); in particular the metadata can include acontent classification. In some aspects, the metadata does not includecontent information desired for encoding operations. In such cases thepreprocessor 226 can be configured to determine content information anduse the content information for preprocessing operations and/or providescontent information to other components of the transcoder 200, e.g., thedecoder 228. In some aspects, the preprocessor 226 can use such contentinformation to influence GOP partitioning, determine appropriate type offiltering, and/or determine encoding parameters that are communicated toan encoder.

FIG. 6 shows an illustrative example of various process blocks that canbe included in the preprocessor 226 and illustrates processing that canbe performed by the preprocessor 226. In this example, the preprocessor226 receives metadata and video 222, 224 and provides output data 614comprising (processed) metadata and video to the encoder 228. Typically,there are three types of video that may be received. First, the receivedvideo can be progressive video, where no deinterlacing is required.Second, the video data can be telecined video, interlaced videoconverted from 24 fps movie sequences, in which case the video. Third,the video can be non-telecined interlaced video. Preprocessor 226 canprocess these types of video as described below.

At block 601, the preprocessor 226 determines if the received video data222, 224 is progressive video. In some cases, this can be determinedfrom the metadata if the metadata contains such information, or byprocessing of the video data itself. For example, an inverse telecineprocess, described below, can determine if the received video 222 isprogressive video. If it is, the process proceeds to block 607 wherefiltering (e.g., denoiser) operations are performed on the video toreduce noise, such as white Gaussian noise. If the video data 222, 224is not progressive video, at block 601 the process proceeds to block 604to a phase detector 604.

Phase detector 604 distinguishes between video that originated in atelecine and that which began in a standard broadcast format. If thedecision is made that the video was telecined (the YES decision pathexiting phase detector 604), the telecined video is returned to itsoriginal format in inverse telecine 606. Redundant frames are identifiedand eliminated and fields derived from the same video frame are rewoveninto a complete image. Since the sequence of reconstructed film imageswere photographically recorded at regular intervals of 1/24 of a second,the motion estimation process performed in a GOP partitioner 612 or thedecoder 228 is more accurate using the inverse telecined images ratherthan the telecined data, which has an irregular time base.

In one aspect, the phase detector 604 makes certain decisions afterreceipt of a video frame. These decisions include: (i) whether thepresent video from a telecine output and the 3:2 pull down phase is oneof the five phases P₀, P₁, P₂, P₃, and P₄ shown in FIG. 38; and (ii) thevideo was generated as conventional NTSC. That decision is denoted asphase P₅. These decisions appear as outputs of phase detector 604 shownin FIG. 2. The path from phase detector 604 labeled “YES” actuates theinverse telecine 606, indicating that it has been provided with thecorrect pull down phase so that it can sort out the fields that wereformed from the same photographic image and combine them. The path fromphase detector 604 labeled “NO” similarly actuates the deinterlacer 605to separate an apparent NTSC frame into fields for optimal processing.The phase detector 604 can continuously analyze video frames thatbecause different types of video may be received at any time. As anexemplary, video conforming to the NTSC standard may be inserted intothe video as a commercial. After inverse telecine, the resultingprogressive video is sent to a denoiser (filter) 607 which can be usedto reduce white Gaussian noise.

When conventional NTSC video is recognized (the NO path from phasedetector 601), it is transmitted to deinterlacer 605 for compression.The deinterlacer 605 transforms the interlaced fields to progressivevideo, and denoising operations can then be performed on the progressivevideo. One illustrative example of deinterlacing processing is describedbelow.

Traditional analog video devices like televisions render video in aninterlaced manner, i.e., such devices transmit even-numbered scan lines(even field), and odd-numbered scan lines (odd field). From the signalsampling point of view, this is equivalent to a spatio-temporalsubsampling in a pattern described by:

$\begin{matrix}{{F\left( {x,y,n} \right)} = \left\{ \begin{matrix}{{\Theta\left( {x,y,n} \right)},{{{if}\mspace{14mu} y\;{mod}\; 2} = {0\mspace{14mu}{for}\mspace{14mu}{even}\mspace{14mu}{fields}}},} \\{{\Theta\left( {x,y,n} \right)},{{{if}\mspace{14mu} y\;{mod}\; 2} = {1\mspace{14mu}{for}\mspace{14mu}{odd}\mspace{14mu}{fields}}},} \\{{Erasure},{otherwise},}\end{matrix} \right.} & \lbrack 1\rbrack\end{matrix}$where Θ stands for the original frame picture, F stands for theinterlaced field, and (x, y, n) represents the horizontal, vertical, andtemporal position of a pixel respectively.

Without loss of generality, it can be assumed n=0 is an even fieldthroughout this disclosure so that Equation 1 above is simplified as

$\begin{matrix}{{F\left( {x,y,n} \right)} = \left\{ \begin{matrix}{{\Theta\left( {x,y,n} \right)},{{{if}\mspace{14mu} y\;{mod}\; 2} = {n\;{mod}\; 2}},} \\{{Erasure},{otherwise},}\end{matrix} \right.} & \lbrack 2\rbrack\end{matrix}$

Since decimation is not conducted in the horizontal dimension, thesub-sampling pattern can be depicted in the next n˜y coordinate.

The goal of a deinterlacer is to transform interlaced video (a sequenceof fields) into non-interlaced progressive frames (a sequence offrames). In other words, interpolate even and odd fields to “recover” orgenerate full-frame pictures. This can be represented by Equation 3:

$\begin{matrix}{{F_{o}\left( {x,y,n} \right)} = \left\{ \begin{matrix}{{F\left( {x,y,n} \right)},{{y\;{mod}\; 2} = {n\;{mod}\; 2}},} \\{{F_{i}\left( {x,y,n} \right)},{otherwise},}\end{matrix} \right.} & \lbrack 3\rbrack\end{matrix}$where F_(i) represent deinterlacing results for missing pixels.

FIG. 40 is a block diagram illustrating certain aspects of an aspect ofa deinterlacer 605 that uses Wmed filtering and motion estimation togenerate a progressive frame from interlaced multimedia data. The upperpart of FIG. 40 shows a motion intensity map 4002 that can be generatedusing information from a current field, two previous fields (PP Fieldand P Field), and two subsequent fields (Next Field and Next Nextfield). The motion intensity map 4002 categorizes, or partitions, thecurrent frame into two or more different motion levels, and can begenerated by spatio-temporal filtering, described in further detailhereinbelow. In some aspects, the motion intensity map 4002 is generatedto identify static areas, slow-motion areas, and fast-motion areas, asdescribed in reference to Equations 4-8 below. A spatio-temporal filter,e.g., Wmed filter 4004, filters the interlaced multimedia data usingcriteria based on the motion intensity map, and produces aspatio-temporal provisional deinterlaced frame. In some aspects, theWmed filtering process involves a horizontal a neighborhood of [−1, 1],a vertical neighborhood of [−3, 3], and a temporal neighborhood of fiveadjacent fields, which are represented by the five fields (PP Field, PField, Current Field, Next Field, Next Next Field) illustrated in FIG.40, with Z⁻¹ representing a delay of one field. Relative to the CurrentField, the Next Field and the P Field are non-parity fields and the PPField and the Next Next Field are parity fields. The “neighborhood” usedfor spatio-temporal filtering refers to the spatial and temporallocation of fields and pixels actually used during the filteringoperation, and can be illustrated as an “aperture” as shown, forexample, in FIGS. 6 and 7.

The deinterlacer 605 can also include a denoiser (denoising filter) 4006configured to filter the spatio-temporal provisional deinterlaced framegenerated by the Wmed filter 4004. Denoising the spatio-temporalprovisional deinterlaced frame makes the subsequent motion searchprocess more accurate especially if the source interlaced multimediadata sequence is contaminated by white noise. It can also at leastpartly remove alias between even and odd rows in a Wmed picture. Thedenoiser 4006 can be implemented as a variety of filters including awavelet shrinkage and wavelet Wiener filter based denoiser. A denoisercan be used to remove noise from the candidate Wmed frame before it isfurther processed using motion compensation information, and can removenoise that is present in the Wmed frame and retain the signal presentregardless of the signal's frequency content. Various types of denoisingfilters can be used, including wavelet filters. Wavelets are a class offunctions used to localize a given signal in both space and scalingdomains. The fundamental idea behind wavelets is to analyze the signalat different scales or resolutions such that small changes in thewavelet representation produce a correspondingly small change in theoriginal signal.

A wavelet shrinkage or a wavelet Wiener filter can be also be applied asthe denoiser. Wavelet shrinkage consists of a wavelet transformation ofthe noisy signal, followed by a shrinking of the small waveletcoefficients to zero (or smaller value), while leaving the largecoefficients unaffected. Finally, an inverse transformation is performedto acquire the estimated signal.

The denoising filtering boosts the accuracy of motion compensation innoisy environments. Wavelet shrinkage denoising can involve shrinking inthe wavelet transform domain, and typically comprises three steps: alinear forward wavelet transform, a nonlinear shrinkage denoising, and alinear inverse wavelet transform. The Wiener filter is a MSE-optimallinear filter which can be used to improve images degraded by additivenoise and blurring. Such filters are generally known in the art and aredescribed, for example, in “Ideal spatial adaptation by waveletshrinkage,” referenced above, and by S. P. Ghael, A. M. Sayeed, and R.G. Baraniuk, “Improvement Wavelet denoising via empirical Wienerfiltering,” Proceedings of SPIE, vol 3169, pp. 389-399, San Diego, July1997, which is expressly incorporated by reference herein in itsentirety.

In some aspects, a denoising filter is based on an aspect of a (4, 2)bi-orthogonal cubic B-spline wavelet filter. One such filter can bedefined by the following forward and inverse transforms:

$\begin{matrix}{{{h(z)} = {\frac{3}{4} + {\frac{1}{2}\left( {z + z^{- 1}} \right)} + {\frac{1}{8}\left( {z + z^{- 2}} \right)\mspace{14mu}\left( {{forward}\mspace{14mu}{transform}} \right)}}}{and}} & \lbrack 4\rbrack \\\begin{matrix}{{g(z)} = {{\frac{5}{4}z^{- 1}} - {\frac{5}{32}\left( {1 + z^{- 2}} \right)} -}} \\{{\frac{3}{8}\left( {z + z^{- 3}} \right)} - {\frac{3}{32}\left( {z^{2} + z^{- 4}} \right)\mspace{14mu}\left( {{inverse}\mspace{14mu}{transform}} \right)}}\end{matrix} & \lbrack 5\rbrack\end{matrix}$

Application of a denoising filter can increase the accuracy of motioncompensation in a noisy environment. Implementations of such filters arefurther described in “Ideal spatial adaptation by wavelet shrinkage,” D.L. Donoho and I. M. Johnstone, Biometrika, vol. 8, pp. 425-455, 1994,which is expressly incorporated by reference herein in its entirety.

The bottom part of FIG. 40 illustrates an aspect for determining motioninformation (e.g., motion vector candidates, motion estimation, motioncompensation) of interlaced multimedia data. In particular, FIG. 40illustrates a motion estimation and motion compensation scheme that isused to generate a motion compensated provisional progressive frame ofthe selected frame, and then combined with the Wmed provisional frame toform a resulting “final” progressive frame, shown as deinterlacedcurrent frame 4014. In some aspects, motion vector (“MV”) candidates (orestimates) of the interlaced multimedia data are provided to thedeinterlacer from external motion estimators and used to provide astarting point for bi-directional motion estimator and compensator(“ME/MC”) 4018. In some aspects, a MV candidate selector 4022 usespreviously determined MV's for neighboring blocks for Mv candidates ofthe blocks being processed, such as the Mvs of previous processedblocks, for example blocks in a deinterlaced previous frame 4020. Themotion compensation can be done bi-directional, based on the previousdeinterlaced frame 70 and a next (e.g., future) Wmed frame 4008. Acurrent Wmed frame 4010 and a motion compensated (“MC”) current frame4016 are merged, or combined, by a combiner 4012. A resultingdeinterlaced current frame 4014, now a progressive frame, is providedback to the ME/MC 4018 to be used as a deinterlaced previous frame 4020and also communicated external to the deinterlacer 605 for subsequentprocessing.

It is possible to decouple deinterlacing prediction schemes comprisinginter-field interpolation from intra-field interpolation with a Wmed+MCdeinterlacing scheme. In other words, the spatio-temporal Wmed filteringcan be used mainly for intra-field interpolation purposes, whileinter-field interpolation can be performed during motion compensation.This reduces the peak signal-to-noise ratio of the Wmed result, but thevisual quality after motion compensation is applied is more pleasing,because bad pixels from inaccurate inter-field prediction mode decisionswill be removed from the Wmed filtering process.

After the appropriate inverse telecine or deinterlacing processing, atblock 608 the progressive video is processed for alias suppressing andresampling (e.g., resizing). In some resampling aspects, a poly-phaseresampler is implemented for picture size resizing. In one example ofdownsampling, the ratio between the original and the resized picture canbe p/q, where p and q are relatively prime integers. The total number ofphases is p. The cutoff frequency of the poly-phase filter in someaspects is 0.6 for resizing factors around 0.5. The cutoff frequencydoes not exactly match the resizing ratio in order to boost thehigh-frequency response of the resized sequence. This inevitably allowssome aliasing. However, it is well-known that human eyes prefer sharpbut a little aliased pictures to blurry and alias-free pictures.

FIG. 41 illustrates an example of poly-phase resampling, showing thephases if the resizing ration is ¾. The cutoff frequency illustrated inFIG. 41 is ¾ also. Original pixels are illustrated in the above figurewith vertical axes. A sinc function is also drawn centered around theaxes to represent the filter waveform. Because we choose the cutofffrequency was chosen to be exactly the same as the resampling ration,the zeros of the sinc function overlap the position of the pixels afterresizing, illustrated in FIG. 41 with crosses. To find a pixel valueafter resizing, the contribution can be summed up from the originalpixels as shown in the following equation:

$\begin{matrix}{{v(x)} = {\sum\limits_{i = {- \infty}}^{\infty}{{u(i)} \times \sin\;{c\left( {\pi\;{f_{c}\left( {i - x} \right)}} \right)}}}} & \lbrack 6\rbrack\end{matrix}$where f_(c) is the cutoff frequency. The above 1-D poly-phase filter canbe applied to both the horizontal dimension and the vertical dimension.

Another aspect of resampling (resizing) is accounting for overscan. Inan NTSC television signal, an image has 486 scan lines, and in digitalvideo could have 720 pixels on each scan line. However, not all of theentire image is visible on the television due to mismatches between thesize and the screen format. The part of the image that is not visible iscalled overscan.

To help broadcasters put useful information in the area visible by asmany televisions as possible, the Society of Motion Picture & TelevisionEngineers (SMPTE) defined specific sizes of the action frame called thesafe action area and the safe title area. See SMPTE recommended practiceRP 27.3-1989 on Specifications for Safe Action and Safe Title Areas TestPattern for Television Systems. The safe action area is defined by theSMPTE as the area in which “all significant action must take place.” Thesafe title area is defined as the area where “all the useful informationcan be confined to ensure visibility on the majority of home televisionreceivers.”

For example, referring to FIG. 25, the safe action area 2510 occupiesthe center 90% of the screen, giving a 5% border all around. The safetitle area 2505 occupies the center 80% of the screen, giving a 10%border. Referring now to FIG. 26, because the safe title area is sosmall, to add more contents in the image, some stations will put textinto the safe action area, which is inside the white rectangular window2615.

Usually black borders may be seen in the overscan. For example, in FIG.26, black borders appear at the upper side 2620 and lower side 2625 ofthe image. These black borders can be removed in the overscan, becauseH.264 video uses boundary extension in motion estimation. Extended blackborders can increase the residual. Conservatively, the boundary can becut by 2%, and then do the resizing. The filters for resizing can begenerated accordingly. Truncation is performed to remove the overscanbefore poly-phase downsampling.

Referring again to FIG. 6, the progressive video then proceeds to block610 where deblocker and deringing operations are performed. Two types ofartifacts, “blocking” and “ringing,” commonly occur in video compressionapplications. Blocking artifacts occur because compression algorithmsdivide each frame into blocks (e.g., 8×8 blocks). Each block isreconstructed with some small errors, and the errors at the edges of ablock often contrast with the errors at the edges of neighboring blocks,making block boundaries visible. In contrast, ringing artifacts appearas distortions around the edges of image features. Ringing artifactsoccur because the encoder discards too much information in quantizingthe high-frequency DCT coefficients. In some illustrative examples, bothdeblocking and deringing can use low-pass FIR (finite impulse response)filters to hide these visible artifacts.

In one example of deblocking processing, a deblocking filter can beapplied to all the 4×4 block edges of a frame, except edges at theboundary of the frame and any edges for which the deblocking filterprocess is disabled. This filtering process shall be performed on amacroblock basis after the completion of the frame construction processwith all macroblocks in a frame processed in order of increasingmacroblock addresses. For each macroblock, vertical edges are filteredfirst, from left to right, and then horizontal edges are filtered fromtop to bottom. The luma deblocking filter process is performed on four16-sample edges and the deblocking filter process for each chromacomponent is performed on two 8-sample edges, for the horizontaldirection and for the vertical direction, as shown in FIG. 39. Samplevalues above and to the left of the current macroblock that may havealready been modified by the deblocking process operation on previousmacroblocks shall be used as input to the deblocking filter process onthe current macroblock and may be further modified during the filteringof the current macroblock. Sample values modified during filtering ofvertical edges can be used as input for the filtering of the horizontaledges for the same macroblock. A deblocking process can be invoked forthe luma and chroma components separately.

In an example of deringing processing, a 2-D filter can be adaptivelyapplied to smooth out areas near edges. Edge pixels undergo little or nofiltering in order to avoid blurring.

GOP Partitioner

After deblocking and deringing, the progressive video is processed by aGOP partitioner 612. GOP positioning can include detecting shot changes,generating complexity maps (e.g., temporal, spatial bandwidth maps), andadaptive GOP partitioning. These are each described below.

A. Scene Change Detection

Shot detection relates to determining when a frame in a group ofpictures (GOP) exhibits data that indicates a scene change has occurred.Generally, within a GOP, the frames may have no significant changes inany two or three (or more) adjacent frames, or there may be slowchanges, or fast changes. Of course, these scene change classificationscan be further broken down to a greater level of changes depending on aspecific application, if necessary.

Detecting shot or scene changes is important for efficient encoding ofvideo. Typically, when a GOP is not changing significantly, an I-frameat the beginning of the GOP is followed by a number of predictive framescan sufficiently encode the video so that subsequent decoding anddisplay of the video is visually acceptable. However, when a scene ischanging, either abruptly or slowly, additional I-frames and lesspredictive encoding (P-frames and B-frames) may be necessary to producesubsequently decoded visually acceptable results.

Shot detection and encoding systems and methods that improve theperformance of existing encoding systems are described below. Suchaspects can be implemented in the GOP partitioner 612 of thepreprocessor 226 (FIG. 7), or included in an encoder device that mayoperate with or without a preprocessor. Such aspects utilizes statistics(or metrics) that include statistical comparisons between adjacentframes of video data to determine if an abrupt scene change occurred, ascene is slowly changing, or if there are camera flashlights in thescene which can make video encoding especially complex. The statisticscan be obtained from a preprocessor and then sent to an encoding device,or they can be generated in an encoding device (e.g., by a processorconfigured to perform motion compensation). The resulting statistics aidscene change detection decision. In a system that does transcoding,often a suitable preprocessor or configurable processor exists. If thepreprocessor perform motion-compensation aided deinterlacing, the motioncompensation statistics are available and ready to use. In such systems,a shot detection algorithm may slightly increases system complexity.

The illustrative example of a shot detector described herein only needsto utilize statistics from a previous frame, a current frame, and a nextframe, and accordingly has very low latency. The shot detectordifferentiates several different types of shot events, including abruptscene change, cross-fading and other slow scene change, and cameraflashlight. By determining different type of shot events with differentstrategies in the encoder, the encoding efficiency and visual quality isenhanced.

Scene change detection can be used for any video coding system for it tointelligently conserve bits by inserting an I-frame at a fixed interval.In some aspects, the content information obtained by the preprocessor(e.g., either incorporated in metadata or calculated by the preprocessor226) can be used for scene change detection. For example, depending onthe content information, threshold values and other criteria describedbelow may be dynamically adjusted for different types of video content.

Video encoding usually operates on a structured group of pictures (GOP).A GOP normally starts with an intra-coded frame (I-frame), followed by aseries of P (predictive) or B (bi-directional) frames. Typically, anI-frame can store all the data required to display the frame, a B-framerelies on data in the preceding and following frames (e.g., onlycontaining data changed from the preceding frame or is different fromdata in the next frame), and a P-frame contains data that has changedfrom the preceding frame. In common usage, I-frames are interspersedwith P-frames and B-frames in encoded video. In terms of size (e.g.,number of bits used to encode the frame), I-frames are typically muchlarger than P-frames, which in turn are larger than B-frames. Forefficient encoding, transmission and decoding processing, the length ofa GOP should be long enough to reduce the efficient loss from bigI-frames, and short enough to fight mismatch between encoder anddecoder, or channel impairment. In addition, macro blocks (MB) in Pframes can be intra coded for the same reason.

Scene change detection can be used for a video encoder to determine aproper GOP length and insert I-frames based on the GOP length, insteadof inserting an often unneeded I-frame at a fixed interval. In apractical streaming video system, the communication channel is usuallyimpaired by bit errors or packet losses. Where to place I frames or IMBs may significantly impact decoded video quality and viewingexperience. One encoding scheme is to use intra-coded frames forpictures or portions of pictures that have significant change fromcollocated previous pictures or picture portions. Normally these regionscannot be predicted effectively and efficiently with motion estimation,and encoding can be done more efficiently if such regions are exemptedfrom inter-frame coding techniques (e.g., encoding using B-frames andP-frames). In the context of channel impairment, those regions arelikely to suffer from error propagation, which can be reduced oreliminated (or nearly so) by intra-frame encoding.

Portions of the GOP video can be classified into two or more categories,where each region can have different intra-frame encoding criteria thatmay depend on the particular implementation. As an example, the videocan be classified into three categories: abrupt scene changes,cross-fading and other slow scene changes, and camera flashlights.

Abrupt scene changes includes frames that are significantly differentfrom the previous frame, usually caused by a camera operation. Since thecontent of these frames is different from that of the previous frame,the abrupt scene change frames should be encoded as I frames.

Cross-fading and other slow scene changes includes slow switching ofscenes, usually caused by computer processing of camera shots. Gradualblending of two different scenes may look more pleasing to human eyes,but poses a challenge to video coding. Motion compensation cannot reducethe bitrate of those frames effectively, and more intra MBs can beupdated for these frames.

Camera flashlights, or camera flash events, occur when the content of aframe includes camera flashes. Such flashes are relatively short induration (e.g., one frame) and extremely bright such that the pixels ina frame portraying the flashes exhibit unusually high luminance relativeto a corresponding area on an adjacent frame. Camera flashlights shiftthe luminance of a picture suddenly and swiftly. Usually the duration ofa camera flashlight is shorter than the temporal masking duration of thehuman vision system (HVS), which is typically defined to be 44 ms. Humaneyes are not sensitive to the quality of these short bursts ofbrightness and therefore they can be encoded coarsely. Because theflashlight frames cannot be handled effectively with motion compensationand they are bad prediction candidate for future frames, coarse encodingof these frames does not reduce the encoding efficiency of futureframes. Scenes classified as flashlights should not be used to predictother frames because of the “artificial” high luminance, and otherframes cannot effectively be used to predict these frames for the samereason. Once identified, these frames can be taken out because they canrequire a relatively high amount of processing. One option is to removethe camera flashlight frames and encode a DC coefficient in their place;such a solution is simple, computationally fast and saves many bits.

When any of the above categories if frames are detected, a shot event isdeclared. Shot detection is not only useful to improve encoding quality,it can also aid in identifying video content searching and indexing. Oneillustrative aspect of a scene detection process is describedhereinbelow. In this example, a shot detection process first calculatesinformation, or metrics, for a selected frame being processed for shotdetection. The metrics can include information from bi-directionalmotion estimation and compensation processing of the video, and otherluminance-based metrics.

To perform bi-directional motion estimation/compensation, a videosequence can be preprocessed with a bi-directional motion compensatorthat matches every 8×8 block of the current frame with blocks in two ofthe frames most adjacent neighboring frames, one in the past, and one inthe future. The motion compensator produces motion vectors anddifference metrics for every block. FIG. 29 is an illustration whichshows an example of matching pixels of a current frame C to a past frameP and a future (or next) frame N, and depicts motion vectors to thematched pixels (past motion vector MV_(P) and future motion vectorMV_(N). A general description of bi-directional motion vector generationand related encoding is generally described hereinbelow in reference toFIG. 32.

After determining bi-directional motion information (e.g., motioninformation which identifies MBs (best matched) in correspondingadjacent frames, additional metrics can be generated (e.g., by a motioncompensator in the GOP partitioner 612 or another suitable component) byvarious comparisons of the current frame to the next frame and theprevious frame. The motion compensator can produce a difference metricfor every block. The difference metric can be a sum of square difference(SSD) or a sum of absolute difference (SAD). Without loss of generality,here SAD is used as an example.

For every frame, a SAD ratio, also referred to as a “contrast ratio,” iscalculated as below:

$\begin{matrix}{\gamma = \frac{ɛ + {SAD}_{P}}{ɛ + {SAD}_{N}}} & \lbrack 6\rbrack\end{matrix}$where SAD_(P) and SAD_(N) are the sum of absolute differences of theforward and the backward difference metric, respectively. It should benoted that the denominator contains a small positive number ε to preventthe “divide-by-zero” error. The nominator also contains an ε to balancethe effect of the unity in the denominator. For example, if the previousframe, the current frame, and the next frame are identical, motionsearch should yield SAD_(P)=SAD_(N)=0. In this case, the abovecalculation generators γ=1 instead of 0 or infinity.

A luminance histogram can be calculated for every frame. Typically themultimedia images have a luminance depth (e.g., number of “bins”) ofeight bits. The luminance depth used for calculating the luminancehistogram according to some aspects can be set to 16 to obtain thehistogram. In other aspects, the luminance depth can be set to anappropriate number which may depend upon the type of data beingprocessed, the computational power available, or other predeterminedcriteria. In some aspects, the luminance depth can be set dynamicallybased on a calculated or received metric, such as the content of thedata.

The equation below illustrates one example of calculating a luminancehistogram difference (lambda):

$\begin{matrix}{\lambda = {\sum\limits_{i = 1}^{16}{{{N_{Pi} - N_{Ci}}}/N}}} & \lbrack 7\rbrack\end{matrix}$where N_(Pi) is the number of blocks in the i^(th) bin for the previousframe, and N_(Ci) is the number of blocks in the i^(th) bin for thecurrent frame, and N is the total number of blocks in a frame. If theluminance histogram difference of the previous and the current frame arecompletely dissimilar (or disjoint), then λ=2.

Using this information, a frame difference metric (D) is calculated asfollows:

$\begin{matrix}{D = {\frac{\gamma_{C}}{\gamma_{P}} + {A\;{\lambda\left( {{2\;\lambda} + 1} \right)}}}} & \lbrack 8\rbrack\end{matrix}$where A is a constant chosen by application,

${\gamma_{C} = \frac{ɛ + {SAD}_{P}}{ɛ + {SAD}_{N}}},{{{and}\mspace{14mu}\gamma_{P}} = {\frac{ɛ + {SAD}_{PP}}{ɛ + {SAD}_{N}}.}}$

The selected (current) frame is classified as an abrupt scene changeframe if the frame difference metric meets the criterion shown inEquation 9:

$\begin{matrix}{D = {{\frac{\gamma_{C}}{\gamma_{P}} + {A\;{\lambda\left( {{2\;\lambda} + 1} \right)}}} \geq T_{1}}} & \lbrack 9\rbrack\end{matrix}$where A is a constant chosen by application, and T₁ is a threshold.

In one example simulation shows, setting A=1, and T₁=5 achieve gooddetection performance. If the current frame is an abrupt scene changeframe, then γ_(C) should be large and γ_(P) should be small. The ratioγ_(C)/γ_(P) can be used instead of γ_(C) alone so that the metric isnormalized to the activity level of the context.

It should be noted that the above criterion uses the luminance histogramdifference lambda (λ) in a non-linear way. FIG. 16 illustrates λ*(2λ+1)is a convex function. When λ is small (e.g., close to zero), it isbarely preemphasis. The larger λ becomes, the more emphasis is conductedby the function. With this pre-emphasis, for any λ larger than 1.4, anabrupt scene change is detected if the threshold T₁ is set at 5.

The current frame is determined to be a cross-fading or slow scenechange if the scene strength metric D meets the criterion shown inEquation 5:T₂≦D<T₁  [10]for a certain number of continuous frames, where T₁ is the samethreshold used above and T₂ is another threshold value.

A flashlight event usually causes the luminance histogram to shift tobrighter side. In this illustrative aspect camera, the luminancehistogram statistics are used to determine if the current framecomprises camera flashlights. A shot detection process can determine ifthe luminance of the current frame minus is greater than the luminanceof the previous frame by a certain threshold T₃, and the luminance ofthe current frame is greater than the luminance of the next frame by thethreshold T₃, as shown in Equations 11 and 12:Y _(C) − Y _(P) ≧T ₃  [11]Y _(C) − Y _(N) ≧T ₃  [12]

If the above criterion are not met, the current frame is not classifiedas comprising camera flashlights. If the criterion is met, the shotdetection process determines if a backwards difference metric SAD_(P)and the forward difference metric SAD_(N) are greater than a certainthreshold T4, as illustrated in the Equations below:SAD_(P)≧T₄  [13]SAD_(N)≧T₄  [14]where Y _(C) is the average luminance of the current frame, Y _(P) isthe average luminance of the previous frame, Y _(N) is the averageluminance of the next frame, and SADP and SADN are the forward andbackward difference metrics associated with the current frame.

The shot detection process determines camera flash events by firstdetermining if the luminance of a current frame is greater than theluminance of the previous frame and the luminance of the next frame. Ifnot, the frame is not a camera flash event; but if so it may be. Theshot detection process then can evaluate whether the backwardsdifference metric is greater than a threshold T₃, and if the forwardsdifference metric is greater than a threshold T₄; if both theseconditions are satisfied, the shot detection process classifies thecurrent frame as having camera flashlights. If the criterion is not met,the frame is not classified as any type of shot even, or it can be givena default classification that identifies the encoding to be done on theframe (e.g., drop frame, encode as I-frame).

Some exemplary values for T₁, T₂, T₃, and T₄ are shown above. Typically,these threshold values are selected through testing of a particularimplementation of shot detection. In some aspects, one or more of thethreshold values T₁, T₂, T₃, and T₄ are predetermined and such valuesare incorporated into the shot classifier in the encoding device. Insome aspects, one or more of the threshold values T₁, T₂, T₃, and T₄ canbe set during processing (e.g., dynamically) based on using information(e.g., metadata) supplied to the shot classifier or based on informationcalculated by the shot classifier itself.

Encoding the video using the shot detection information is typicallyperformed in the encoder, but is described here for completeness of theshot detection disclosure. Referring to FIG. 30, an encoding process 301can use the shot detection information to encode the video based uponthe detected shots in the sequence of frames. Process 301 proceeds toblock 303, and checks to see if the current frame is classified as anabrupt scene change. If so, at block 305 the current frame can beencoded as an I-frame and a GOP boundary can be determined. If not,process 301 proceeds to block 307; if the current frame is classified asa portion of a slowly changing scene at block 309 the current frame, andother frames in the slow changing scene can be encoded as a predictiveframe (e.g., P-frame or B-frame). Process 301 then proceeds to block 311where it checks if the current frame is be classified as a flashlightscene comprising camera flashes. If so, at block 313 the frame can beidentified for special processing, for example, removal or encoding a DCcoefficient for the frame; if not, no classification of the currentframe was made and the current frame can be encoded in accordance withother criteria, encoded as an I-frame, or dropped.

In the above-described aspect, the amount of difference between theframe to be compressed and its adjacent two frames is indicated by aframe difference metric D. If a significant amount of a one-wayluminance change is detected, it signifies a cross-fade effect in theframe. The more prominent the cross-fade is, the more gain may beachieved by using B-frames. In some aspects, a modified frame differencemetric is used as shown in the equation below:

$\begin{matrix}{D_{1} = \left\{ \begin{matrix}{{\left( {1 - \alpha + {2\alpha\frac{{\mathbb{d}_{P}{- \mathbb{d}_{N}}}}{\mathbb{d}_{P}{+ \mathbb{d}_{N}}}}} \right) \times D},} & \begin{matrix}{{{{{if}\mspace{11mu} Y_{P}} - \Delta} \geq Y_{C} \geq {Y_{N} + {\Delta\mspace{14mu}{or}}}}\;} \\{{{Y_{P} + \Delta} \leq Y_{C} \leq {Y_{N} - \Delta}},}\end{matrix} \\{D,} & {{otherwise},}\end{matrix} \right.} & \lbrack 15\rbrack\end{matrix}$where d_(P)=|Y_(C)−Y_(P)| and d_(N)=|Y_(C)−Y_(N)| are the lumadifference between the current frame and the previous frame, and theluma difference between the current frame and the next frame,respectively, Δ represents a constant that can be determined in normalexperimentation as it can depend on the implementation, and α is aweighting variable having a value between 0 and 1.B. Bandwidth Map Generation

The preprocessor 226 (FIG. 6) can also be configured to generate abandwidth map which can be used for encoding the multimedia data. Insome aspects, a content classification module 712 in the encoder 228(FIG. 7) generates the bandwidth map instead.

Human visual quality V can be a function of both encoding complexity Cand allocated bits B (also referred to as bandwidth). FIG. 15 is a graphillustrating this relationship. It should be noted that the encodingcomplexity metric C considers spatial and temporal frequencies from thehuman vision point of view. For distortions more sensitive to humaneyes, the complexity value is correspondingly higher. It can typicallybe assume that V is monotonically decreasing in C, and monotonicallyincreasing in B.

To achieve constant visual quality, a bandwidth (B_(i)) is assigned tothe i^(th) object (frame or MB) to be encoded that satisfies thecriteria expressed in the two equations immediately below:

$\begin{matrix}{B_{i} = {B\left( {C_{i},V} \right)}} & \lbrack 16\rbrack \\{B = {\sum\limits_{i}B_{i}}} & \lbrack 17\rbrack\end{matrix}$

In the two equations immediately above, C_(i) is the encoding complexityof the i^(th) object, B is the total available bandwidth, and V is theachieved visual quality for an object. Human visual quality is difficultto formulate as an equation. Therefore, the above equation set is notprecisely defined. However, if it is assumed that the 3-D model iscontinuous in all variables, bandwidth ratio (B_(i)/B) can be treated asunchanged within the neighborhood of a (C, V) pair. The bandwidth ratioβ_(i) is defined in the equation shown below:β_(i) =B _(i) /B  [18]

Bit allocation can then be defined as expressed in the followingequations:

$\begin{matrix}{\beta_{i} = {\beta\left( C_{i} \right)}} & \; \\{1 = {{\sum\limits_{i}{\beta_{i}\mspace{14mu}{for}\mspace{14mu}\left( {C_{i},V} \right)}} \in {\delta\left( {C_{0},V_{0}} \right)}}} & \lbrack 19\rbrack\end{matrix}$where δ indicates the “neighborhood.”

The encoding complexity is affected by human visual sensitivity, bothspatial and temporal. Girod's human vision model is an example of amodel that can be used to define the spatial complexity. This modelconsiders the local spatial frequency and ambient lighting. Theresulting metric is called D_(csat). At a pre-processing point in theprocess, whether a picture is to be intra-coded or inter-coded is notknown and bandwidth ratios for both are generated. Bits are allocatedaccording to the ratio between β_(INTRA) of different video objects. Forintra-coded pictures, the bandwidth ratio is expressed in the followingequation:β_(INTRA)=β_(0INTRA)log₁₀(1+α_(INTRA) Y ² D _(csat))  [20]

In the equation above, Y is the average luminance component of amacroblock, α_(INTRA) is a weighing factor for the luminance square andD_(csat) term following it, β_(0INTRA) is a normalization factor toguarantee

$1 = {\sum\limits_{i}{\beta_{i}.}}$For example, a value for α_(INTRA)=4 achieves good visual quality.Content information (e.g., a content classification) can be used to setα_(INTRA) to a value that corresponds to a desired good visual qualitylevel for the particular content of the video. In one example, if thevideo content comprises a “talking head” news broadcast, the visualquality level may be set lower because the information image ordisplayable portion of the video may be deemed of less importance thanthe audio portion, and less bits can be allocated to encode the data. Inanother example, if the video content comprises a sporting event,content information may be used to set α_(INTRA) to a value thatcorresponds to a higher visual quality level because the displayedimages may be more important to a viewer, and accordingly more bits canbe allocated to encode the data.

To understand this relationship, it should be noted that bandwidth isallocated logarithmically with encoding complexity. The luminancesquared term Y² reflects the fact that coefficients with largermagnitude use more bits to encode. To prevent the logarithm from gettingnegative values, unity is added to the term in the parenthesis.Logarithms with other bases can also be used.

The temporal complexity is determined by a measure of a frame differencemetric, which measures the difference between two consecutive framestaking into account the amount of motion (e.g., motion vectors) alongwith a frame difference metric such as the sum of the absolutedifferences (SAD).

Bit allocation for inter-coded pictures can consider spatial as well astemporal complexity. This is expressed below:β_(INTER)=β_(0INTER)log₁₀(1+α_(INTER) ·SSD·D _(csat)exp(−γ∥MV _(P) +MV_(N)∥²))  [21]

In the above equation, MV_(P) and MV_(N) are the forward and thebackward motion vectors for the current MB (see FIG. 29). It can benoted that Y² in the intra-coded bandwidth formula is replaced by sum ofsquared differences (SSD). To understand the role of ∥MV_(P)+MV_(N)∥² inthe above equation, note the next characteristics of human visualsystem: areas undergoing smooth, predictable motion (small∥MV_(P)+MV_(N)∥²) attract attention and can be tracked by the eye andtypically cannot tolerate any more distortion than stationary regions.However, areas undergoing fast or unpredictable motion (large∥MV_(P)+MV_(N)∥²) cannot be tracked and can tolerate significantquantization. Experiments show that α_(INTER)=1, γ=0.001 achieves goodvisual quality.

C. Adaptive GOP Partitioning

In another illustrative example of processing that may be performed bythe preprocessor 226, the GOP Partitioner 612 of FIG. 6 can alsoadaptively change the composition of a group of pictures coded together,and is discussed in reference to an example using MPEG2. Some oldervideo compression standards (e.g., MPEG2) does not require that a GOPhave a regular structure, though one can be imposed. The MPEG2 sequencealways begins with an I frame, i.e., one which has been encoded withoutreference to previous pictures. The MPEG2 GOP format is usuallyprearranged at the encoder by fixing the spacing in the GOP of the P orpredictive pictures that follow the I frame. P frames are pictures thathave been in part predicted from previous I or P pictures. The framesbetween the starting I frame, and the succeeding P frames are encoded asB-frames. A “B” frame (B stands for bi-directional) can use the previousand next I or P pictures either individually or simultaneously asreference. The number of bits needed to encode an I frame on the averageexceeds the number of bits needed to encode a P frame; likewise thenumber of bits needed to encode a P frame on the average exceeds thatrequired for a B-frame. A skipped frame, if it is used, would require nobits for its representation.

The concept underlying the use of P and B-frames, and in more recentcompression algorithms, the skipping of frames to reduce the rate of thedata needed to represent the video is the elimination of temporalredundancy. When temporal redundancy is high—i.e., there is littlechange from picture to picture—use of P, B, or skipped picturesefficiently represents the video stream, because I or P pictures decodedearlier are used later as references to decode other P or B pictures.

Adaptive GOP partitioning is based on using this concept adaptively.Differences between frames are quantified and a decision to representthe picture by a I, P, B, or skipped frame is automatically made aftersuitable tests are performed on the quantified differences. An adaptivestructure has advantages not available in a fixed GOP structure. A fixedstructure would ignore the possibility that little change in content hastaken place; an adaptive procedure would allow far more B-frames to beinserted between each I and P, or two P frames, thereby reducing thenumber of bits needed to adequately represent the sequence of frames.Conversely when the change in video content is significant, theefficiency of P frames is greatly reduced because the difference betweenthe predicted and the reference frames is too large. Under theseconditions, matching objects may fall out of the motion search regions,or the similarity between matching objects is reduced due to distortioncaused by changes in camera angle. At that point the P frames or the Iand its adjacent P frame should be chosen to be closer to each other andfewer B-frames should be inserted. A fixed GOP could not make thatadjustment.

In the system disclosed here, these conditions are automatically sensed.The GOP structure is flexible and is made to adapt to these changes incontent. The system evaluates a frame difference metric, which can bethought of as measure of distance between frames, with the same additiveproperties of distance. In concept, given frames F₁, F₂, and F₃ havingthe inter-frame distances d₁₂ and d₂₃, the distance between F₁ and F₃ istaken as being at least d₁₂+d₂₃. Frame assignments are made on the basisof this distance-like metric.

The GOP partitioner operates by assigning picture types to frames asthey are received. The picture type indicates the method of predictionthat may be required in coding each block:

I-pictures are coded without reference to other pictures. Since theystand alone they provide access points in the data stream where decodingcan begin. An I encoding type is assigned to a frame if the “distance”to its predecessor frame exceeds a scene change threshold.

P-pictures can use the previous I or P pictures for motion compensatedprediction. They use blocks in the previous fields or frames that may bedisplaced from the block being predicted as a basis for encoding. Afterthe reference block is subtracted from the block being considered, theresidual block is encoded, typically using the discrete cosine transformfor the elimination of spatial redundancy. A P encoding types isassigned to a frame if the “distance” between it and the last frameassigned to be a P frame exceeds a second threshold, which is typicallyless than the first.

B-frame pictures can use the previous and next P- or I-pictures formotion compensation as described above. A block in a B picture can beforward, backward or bi-directionally predicted; or it could beintra-coded without reference to other frames. In H.264 a referenceblock can be a linear combination of as many as 32 blocks from as manyframes. If the frame cannot be assigned to be an I or P type, it isassigned to be a B type, if the “distance” from it to its immediatepredecessor is greater than a third threshold, which typically is lessthan the second threshold.

If the frame cannot be assigned to become a B-frame encoded, it isassigned to “skip frame” status. This frame can be skipped because it isvirtually a copy of a previous frame.

Evaluating a metric that quantifies the difference between adjacentframes in the display order is the first part of this processing thattakes place. This metric is the distance referred to above; with it,every frame is evaluated for its proper type. Thus, the spacing betweenthe I and adjacent P, or two successive P frames, can be variable.Computing the metric begins by processing the video frames with ablock-based motion compensator, a block being the basic unit of videocompression, composed usually of 16×16 pixels, though other block sizessuch as 8×8, 4×4 and 8×16 are possible. For frames consisting of twodeinterlaced fields, the motion compensation can be done on a fieldbasis, the search for the reference blocks taking place in fields ratherthan frames. For a block in the first field of the current frame aforward reference block is found in fields of the frame that follows it;likewise a backward reference block found in fields of the frame thatimmediately precedes the current field. The current blocks are assembledinto a compensated field. The process continues with the second field ofthe frame. The two compensated fields are combined to form a forward anda backward compensated frame.

For frames created in the inverse telecine 606, the search for referenceblocks is on a frame basis only, since only reconstructed film frames.Two reference blocks and two differences, forward and backward, arefound, leading also to a forward and backward compensated frame. Insummary, the motion compensator produces motion vectors and differencemetrics for every block; but a block is part of a NTSC field in the caseof the output of deinterlacer 605 being processed and is part of a filmframe if the inverse telecine's output is processed. Note that thedifferences in the metric are evaluated between a block in the field orframe being considered and a block that best matches it, either in apreceding field or frame or a field or frame that immediately followsit, depending on whether a forward or backward difference is beingevaluated. Only luminance values enter into this calculation.

The motion compensation step thus generates two sets of differences.These are between blocks of current values of luminance and theluminance values in reference blocks taken from frames that areimmediately ahead and immediately behind the current one in time. Theabsolute value of each forward and each backward difference isdetermined for each pixel and each is separately summed over the entireframe. Both fields are included in the two summations when thedeinterlaced NTSC fields that comprise a frame are processed. In thisway, SAD_(P), and SAD_(N), the summed absolute values of the forward andbackward differences are found.

For every frame a SAD ratio is calculated using the relationship,

$\begin{matrix}{\gamma = \frac{ɛ + {SAD}_{P}}{ɛ + {SAD}_{N}}} & \lbrack 22\rbrack\end{matrix}$

where SAD_(P) and SAD_(N) are the summed absolute values of the forwardand backward differences respectively. A small positive number is addedto the numerator ε to prevent the “divide-by-zero” error. A similar εterm is added to the denominator, further reducing the sensitivity of γwhen either SAD_(P) or SAD_(N) is close to zero.

In an alternate aspect, the difference can be the SSD, the sum ofsquared differences, and SAD, the sum of absolute differences, or theSATD, in which the blocks of pixel values are transformed by applyingthe two dimensional Discrete Cosine Transform to them before differencesin block elements are taken. The sums are evaluated over the area ofactive video, though a smaller area may be used in other aspects.

The luminance histogram of every frame as received (non-motioncompensated) is also computed. The histogram operates on the DCcoefficient, i.e., the (0,0) coefficient, in the 16×16 array ofcoefficients that is the result of applying the two dimensional DiscreteCosine Transform to the block of luminance values if it were available.Equivalently the average value of the 256 values of luminance in the16×16 block may be used in the histogram. For images whose luminancedepth is eight bits, the number of bins is set at 16. The next metricevaluates the histogram difference

$\begin{matrix}{\lambda\; = \;{\frac{1}{N}\;{\sum\limits_{i\; = \; 1}^{16}{{N_{Pi} - N_{Ci}}}}}} & \lbrack 23\rbrack\end{matrix}$In the above, N_(Pi) is the number of blocks from the previous frame inthe i^(th) bin, and N_(ci) is the number of blocks from the currentframe that belong in the i^(th) bin, N is the total number of blocks ina frame.

These intermediate results are assembled to form the current framedifference metric as

$\begin{matrix}{{D = {\frac{\gamma_{C}}{\gamma_{P}} + {\lambda\left( {{2\lambda} + 1} \right)}}},} & \lbrack 24\rbrack\end{matrix}$

where γ_(C) is the SAD ratio based on the current frame and γ_(P) is theSAD ratio based on the previous frame. If a scene has smooth motion andits luma histogram barely change, then D≈1. If the current framedisplays an abrupt scene change, then γ_(C) will be large and γ_(P)should be small. The ratio γ_(C)/γ_(P) instead of γ_(C) alone is used sothat the metric is normalized to the activity level of the context.

FIG. 42 illustrates the process of assigning compression types toframes. D, the current frame difference defined in Equation 19, is thebasis for decisions made with respect to frame assignments. As decisionblock 4202 indicates, if a frame under consideration is the first in asequence, the decision path marked YES is followed to block 4206,thereby declaring the frame to be an I frame. The accumulated framedifferences is set to zero in block 4208, and the process returns (inblock 4210) to the start block. If the frame being considered is not thefirst frame in a sequence, the path marked NO is followed from block4202 where the decision was made, and in test block 4204 the currentframe difference is tested against the scene change threshold. If thecurrent frame difference is larger than that threshold, the decisionpath marked YES is followed to block 4206, again leading to theassignment of an I-frame.

If the current frame difference is less than the scene change threshold,the NO path is followed to block 4212 where the current frame differenceis added the accumulated frame difference. Continuing through theflowchart at decision block 4214, the accumulated frame difference iscompared with threshold t, which is in general less than the scenechange threshold. If the accumulated frame difference is larger than t,control transfers to block 4216, and the frame is assigned to be a Pframe; the accumulated frame difference is then reset to zero in step4218. If the accumulated frame difference is less than t, controltransfers from block 4214 to block 4220. There the current framedifference is compared with τ, which is less than t. If the currentframe difference is smaller than τ, the frame is assigned to be skippedin block 4222 and then the process returns; if the current framedifference is larger than τ, the frame is assigned to be a B-frame inblock 4226.

Encoder

Referring back to FIG. 2, the transcoder 200 includes an encoder 228that receives processed metadata and raw video from the preprocessor226. The metadata can include any information originally received in thesource video 104 and any information calculated by the preprocessor 226.The encoder 228 includes a first pass encoder 230, a second pass encoder232, and a re-encoder 234. The encoder 228 also receives input from thetranscoder control 231 which can provide information (e.g., metadata,error resilience information, content information, encoded bitrateinformation, base-layer and enhancement-layer balance information, andquantization information) from the second pass encoder 232 to the firstpass encoder 230, the re-encoder 234, as well as the preprocessor 226.The encoder 228 encodes the received video using content informationreceived from the preprocessor 226 and/or content information that isgenerated by the encoder 228 itself, for example, by the contentclassification module 712 (FIG. 7).

FIG. 7 illustrates a block diagram of functional modules that can beincluded in an exemplary two-pass encoder that may be used for theencoder 228 illustrated in FIG. 2. Various aspects of the functionalmodules are shown in FIG. 7, although FIG. 7 and the description hereindoes not necessarily address all functionality that can be incorporatedinto an encoder. Accordingly, certain aspects of the functional modulesare described below following the discussion of base and enhancementlayer encoding below.

Base Layer and Enhancement Layer Encoding

The encoder 228 can be a SNR scalable encoder, which can encode the rawvideo and the metadata from the preprocessor 226 into a first group ofencoded data, also referred to herein as a base layer, and one or moreadditional groups of encoded data, also referred to herein asenhancement layers. An encoding algorithm generates base layer andenhancement layer coefficients which, when decoded, may be combined atthe decoder when both layers are available for decoding. When bothlayers are not available, the encoding of the base layer allows it to bedecoded as a single layer.

One aspect of a such a multilayer encoding process is described inreference to FIG. 31. At block 321, an I frame is encoded with entirelyintra-coded macroblocks (intra-coded MBs). In H.264, intra-coded MBs inI frames are encoded with fully exploited spatial prediction, whichprovides a significant amount of coding gain. There are two sub-modes:Intra4×4 and Intra16×16. If the base layer is to take advantage of thecoding gain provided by spatial prediction, then the base layer needs tobe encoded and decoded before encoding and decoding the enhancementlayer. A two pass encoding and decoding of I frames is used. In the baselayer, a base layer quantization parameter QP_(b) affords the transformcoefficients a coarse quantization step size. The pixel-wise differencebetween the original frame and the reconstructed base layer frame willbe encoded at the enhancement layer. The enhancement layer uses aquantization parameter QP_(e) which affords a finer quantization stepsize. Encoding means, such as encoder 228 of FIG. 2 can perform theencoding at block 321.

At block 323, an encoder encodes base layer data and enhancement layerdata for P and/or B-frames in the GOP being processed. Encoding means,such as encoder 228 can perform the encoding at block 323. At block 325,the encoding process checks if there are more P or B-frames to encode.Encoding means, such as SNR scalable encoder 228 can perform act 325. Ifmore P or B-frames remain, step 323 is repeated until all the frames inthe GOP are finished being encoded. P and B-frames are comprised ofinter-coded macroblocks (inter-coded MBs), although there can beintra-coded MB's in P and B-frames as will be discussed below.

In order for a decoder to distinguish between base layer and enhancementlayer data, the encoder 228 encodes overhead information, block 327. Thetypes of overhead information include, for example, data identifying thenumber of layers, data identifying a layer as a base layer, dataidentifying a layer as an enhancement layer, data identifyinginter-relationships between layers (such as, layer 2 is an enhancementlayer for base layer 1, or layer 3 is an enhancement layer forenhancement layer 2), or data identifying a layer as a final enhancementlayer in a string of enhancement layers. The overhead information can becontained in headers connected with the base and/or enhancement layerdata that it pertains to, or contained in separate data messages.Encoding means, such as encoder 228 of FIG. 2 can perform the process atblock 327.

To have single layer decoding, the coefficients of two layers must becombined before inverse quantization. Therefore the coefficients of thetwo layers have to be generated interactively; otherwise this couldintroduce a significant amount of overhead. One reason for the increasedoverhead is that the base layer encoding and the enhancement layerencoding could use different temporal references. An algorithm is neededto generate base layer and enhancement layer coefficients, which can becombined at the decoder before dequantization when both layers areavailable. At the same time, the algorithm should provide for acceptablebase layer video when the enhancement layer is not available or thedecoder decides not to decode the enhancement layer for reasons such as,for example, power savings. The details of an illustrative example ofsuch a process are discussed further below in context of the briefdiscussion of standard predictive coding immediately below.

P-frames (or any inter-coded sections) can exploit temporal redundancybetween a region in a current picture and a best matching predictionregion in a reference picture. The location of the best matchingprediction region in the reference frame can be encoded in a motionvector. The difference between the current region and the best matchingreference prediction region is known as residual error (or predictionerror).

FIG. 32 is an illustration of an example of a P-frame constructionprocess in, for example, MPEG-4. Process 331 is a more detailedillustration of an example process that could take place in block 323 ofFIG. 31. Process 331 includes current picture 333 made up of 5×5macroblocks, where the number of macroblocks in this example isarbitrary. A macroblock is made up of 16×16 pixels. Pixels can bedefined by an 8-bit luminance value (Y) and two 8-bit chrominance values(Cr and Cb). In MPEG, Y, Cr and Cb components can be stored in a 4:2:0format, where the Cr and Cb components are down-sampled by 2 in the Xand the Y directions. Hence, each macroblock would consist of 256 Ycomponents, 64 Cr components and 64 Cb components. Macroblock 335 ofcurrent picture 333 is predicted from reference picture 337 at adifferent time point than current picture 333. A search is made inreference picture 337 to locate best matching macroblock 339 that isclosest, in terms of Y, Cr and Cb values to current macroblock 335 beingencoded. The location of best matching macroblock 339 in referencepicture 337 is encoded in motion vector 341. Reference picture 337 canbe an I-frame or P Frame that a decoder will have reconstructed prior tothe construction of current picture 333. Best matching macroblock 339 issubtracted from current macroblock 335 (a difference for each of the Y,Cr and Cb components is calculated) resulting in residual error 343.Residual error 343 is encoded with 2D Discrete Cosine Transform (DCT)345 and then quantized 347. Quantization 347 can be performed to providespatial compression by, for example, allotting fewer bits to the highfrequency coefficients while allotting more bits to the low frequencycoefficients. The quantized coefficients of residual error 343, alongwith motion vector 341 and reference picture 333 identifyinginformation, are encoded information representing current macroblock335. The encoded information can be stored in memory for future use oroperated on for purposes of, for example, error correction or imageenhancement, or transmitted over network 349.

The encoded quantized coefficients of residual error 343, along withencoded motion vector 341 can be used to reconstruct current macroblock335 in the encoder for use as part of a reference frame for subsequentmotion estimation and compensation. The encoder can emulate theprocedures of a decoder for this P Frame reconstruction. The emulationof the decoder will result in both the encoder and decoder working withthe same reference picture. The reconstruction process, whether done inan encoder, for further inter-coding, or in a decoder, is presentedhere. Reconstruction of a P Frame can be started after the referenceframe (or a portion of a picture or frame that is being referenced) isreconstructed. The encoded quantized coefficients are dequantized 351and then 2D Inverse DCT, or IDCT, 353 is performed resulting in decodedor reconstructed residual error 355. Encoded motion vector 341 isdecoded and used to locate the already reconstructed best matchingmacroblock 357 in the already reconstructed reference picture 337.Reconstructed residual error 355 is then added to reconstructed bestmatching macroblock 357 to form reconstructed macroblock 359.Reconstructed macroblock 359 can be stored in memory, displayedindependently or in a picture with other reconstructed macroblocks, orprocessed further for image enhancement.

B-frames (or any section coded with bi-directional prediction) canexploit temporal redundancy between a region in a current picture and abest matching prediction region in a previous picture and a bestmatching prediction region in a subsequent picture. The subsequent bestmatching prediction region and the previous best matching predictionregion are combined to form a combined bi-directional predicted region.The difference between the current picture region and the best matchingcombined bi-directional prediction region is a residual error (orprediction error). The locations of the best matching prediction regionin the subsequent reference picture and the best matching predictionregion in the previous reference picture can be encoded in two motionvectors.

FIG. 33 illustrates an example of an encoder process for encoding ofbase layer and enhancement layer coefficients that can be performed byencoder 228. The base and enhancement layers are encoded to provide anSNR scalable bitstream. FIG. 33 depicts an example for encoding inter MBresidual error coefficients such as would be done in step 323 of FIG.31. However, similar methods could be used to encode intra MBcoefficients as well. Encoding means such as encoder component 228 ofFIG. 2 can perform the process illustrated in FIG. 33 and step 323 ofFIG. 32. Original (to be encoded) video data 406 (video data comprisesluma and chroma information in this example) is input to a base layerbest matching macroblock loop 302 and an enhancement layer best matchingmacroblock loop 365. The object of both loops 363 and 365 is to minimizethe residual error that is calculated at adders 367 and 369respectively. Loops 363 and 365 can be performed in parallel, as shown,or sequentially. Loops 363 and 365 include logic for searching buffers371 and 373, respectively, which contain reference frames, to identifythe best matching macroblock that minimizes the residual error betweenthe best matching macroblock and original data 361 (buffers 371 and 373can be the same buffer). The residual errors of loop 363 and loop 365will be different since base layer loop 363 will generally utilize acoarser quantization step size (a higher QP value) than the enhancementlayer loop 365. Transform blocks 375 and 377 transform the residualerrors of each loop.

The transformed coefficients are then parsed into base layer andenhancement layer coefficients in selector 379. The parsing of selector379 can take on several forms, as discussed below. One common feature ofthe parsing techniques is that the enhancement layer coefficient,C′_(enh), is calculated such that it is a differential refinement to thebase layer coefficient C′_(base). Calculating the enhancement layer tobe a refinement to the base layer allows a decoder to decode the baselayer coefficient by itself and have a reasonable representation of theimage, or to combine the base and enhancement layer coefficients andhave a refined representation of the image. The coefficients selected byselector 379 are then quantized by quantizers 381 and 383. The quantizedcoefficients {tilde over (C)}′_(base) and {tilde over (C)}′_(enh)(calculated with quantizers 381 and 383 respectively) can be stored inmemory or transmitted over a network to a decoder.

To match the reconstruction of the macroblock in a decoder, dequantizer385 dequantizes the base layer residual error coefficients. Thedequantized residual error coefficients are inverse transformed 387 andadded 389 to the best matching macroblock found in buffer 371, resultingin a reconstructed macroblock that matches what will be reconstructed inthe decoder. Quantizer 383, dequantizer 391, inverse transformer 393,adder 397 and buffer 373 perform similar calculations in enhancementloop 365 as were done in base layer loop 363. In addition, adder 393 isused to combine the dequantized enhancement layer and base layercoefficients used in the reconstruction of the enhancement layer. Theenhancement layer quantizer and dequantizer will generally utilize afiner quantizer step size (a lower QP) than the base layer.

FIGS. 34, 35 and 36 show examples of base layer and enhancement layercoefficient selector processes that can be employed in selector 379 ofFIG. 33. Selecting means such as encoder 228 of FIG. 2 can perform theprocesses depicted in FIGS. 34, 35 and 35. Using FIG. 34 as an example,the transformed coefficients are parsed into base and enhancement layercoefficients as shown in the following equations:

$\begin{matrix}{C_{base}^{\prime} = \left\{ \begin{matrix}{0,} & {{if}\mspace{14mu} C_{base}\mspace{14mu}{and}\mspace{14mu} C_{enh}\mspace{14mu}{are}\mspace{14mu}{opposite}\mspace{14mu}{signs}} \\{{\min\mspace{11mu}\left( {C_{base},C_{enh}} \right)},} & {otherwise}\end{matrix} \right.} & \lbrack 25\rbrack \\{C_{enh}^{\prime} = {C_{enh} - {Q_{b}^{- 1}\left( {Q_{b}\left( C_{base}^{\prime} \right)} \right)}}} & \lbrack 26\rbrack\end{matrix}$where the “min” function can be either a mathematical minimum or aminimum magnitude of the two arguments. Equation 25 is depicted as block401 and Equation 26 is depicted as adder 510 in FIG. 34. In Equation 26,Q_(b) stands for the base layer quantizer 381, and Q_(b) ⁻¹ stands fordequantizer 385 of the base layer. Equation 26 converts the enhancementlayer coefficient into a differential refinement of the base layercoefficient calculated with Equation 25.

FIG. 35 is an illustration of another example of a base layer andenhancement layer coefficient selector 379. In this example the Equation(.) contained in block 405 represents the following:

$\begin{matrix}{C_{base}^{\prime} = \left\{ \begin{matrix}C_{{base},} & {{{if}\mspace{14mu}{{{Q_{b}^{- 1}\;{Q_{b}\left( C_{base} \right)}}\; - C_{enh}}}} < {C_{enh}}} \\{0,} & {otherwise}\end{matrix} \right.} & \lbrack 27\rbrack\end{matrix}$

Adder 407 computes the enhancement layer coefficient as shown in thefollowing two equations:C′ _(enh) =C _(enh) −Q _(b) ⁻¹(Q _(b)(C′ _(base)))  [28]where C′_(base) is given by Equation 27.

FIG. 36 is an illustration of another example of a base layer andenhancement layer selector 379. In this example, the base layercoefficient is unchanged and the enhancement layer is equal to thedifference between the quantized/dequantized base layer coefficient andthe original enhancement layer coefficient.

In addition to the base and enhancement layer residual errorcoefficients, the decoder needs information identifying how MB's areencoded. Encoding means such as encoder component 228 of FIG. 2 canencode overhead information that can include a map of intra-coded andinter-coded portions, such as, for example a MB map where macroblocks(or sub-macroblocks) are identified as being intra-coded or inter-coded(also identifying which type of inter-coding including, for exampleforward, backward or bi-directional) and to which frame(s) inter-codedportions are referenced. In an example aspect, the MB map and base layercoefficients are encoded in the base layer, and the enhancement layercoefficient are encoded in the enhancement layer.

P-frames and B-frames can contain intra-coded MBs as well as inter MBs.It is common for hybrid video encoders to use rate distortion (RD)optimization to decide to encode certain macroblocks in P or B-frames asintra-coded MBs. In order to have single layer decoding whereintra-coded MB's do not depend on enhancement layer inter MB's, anyneighboring inter MBs are not used for spatial prediction of base layerintra-coded MBs. In order to keep the computational complexity unchangedfor the enhancement layer decoding, for the intra-coded MBs in the baselayer P or B-frame, the refinement at the enhancement layer could beskipped.

Intra-coded MBs in P or B-frames require many more bits than inter MBs.For this reason, intra-coded MBs in P or B-frames could be encoded onlyat base layer quality at a higher QP. This will introduce somedeterioration in video quality, but this deterioration should beunnoticeable if it is refined in a later frame with the inter MBcoefficients in the base and enhancement layer as discussed above. Tworeasons make this deterioration unnoticeable. The first is a feature ofthe human visual system (HVS) and the other one is that Inter MBs refineintra MBs. With objects that change position from a first frame to asecond frame, some pixels in the first frame are invisible in the secondframe (to-be-covered information), and some pixels in the second frameare visible for the first time (uncovered information). Human eyes arenot sensitive to the uncovered and to-be-covered visual information. Sofor the uncovered information, even though it is encoded at a lowerquality, the eyes may not tell the difference. If the same informationremains in the following P frame, there will be a high chance that thefollowing P frame at the enhancement layer can refine it because theenhancement layer has lower QP.

Another common technique that introduces intra-coded MBs in P orB-frames is known as Intra Refresh. In this case, some MBs are coded asintra-coded MBs, even though standard R-D optimization would dictatethat they should be Inter-coded MBs. These intra-coded MBs, contained inthe base layer, can be encoded with either QP_(b) or QP_(e). If QP_(e),is used for the base layer, then no refinement is needed at theenhancement layer. If QP_(b), is used for the base layer, thenrefinement may be needed, otherwise at the enhancement layer, the dropof quality will be noticeable. Since inter-coding is more efficient thanintra-coding in the sense of coding efficiency, these refinements at theenhancement layer will be inter-coded. This way, the base layercoefficients will not be used for the enhancement layer. Therefore thequality gets improved at the enhancement layer without introducing newoperations.

B-frames are commonly used in enhancement layers because of the highcompression qualities they offer. However, B-frames may have toreference intra-coded MBs of a P frame. If the pixels of the B-framewere to be encoded at enhancement layer quality, it may require too manybits due to the lower quality of the P frame intra-coded MBs, asdiscussed above. By taking advantage of the qualities of the HVS, asdiscussed above, the B-frame MBs could be encoded at a lower qualitywhen referencing lower quality intra-coded MB's of P frames.

One extreme case of intra-coded MBs in P or B-frames is when all MBs ina P or B-frame are encoded at Intra mode due to the presence of a scenechange in the video being encoded. In this case the whole frame can becoded at the base layer quality and no refinement at the enhancementlayer. If a scene change occurs at a B-frame, and assume that B-framesare only encoded in the enhancement layer, then the B-frame could beencoded at base layer quality or simply dropped. If a scene changeoccurs at a P frame, no changes may be needed, but the P frame could bedropped or encoded at base layer quality. Scalable layer encoding isfurther described in co-pending U.S. patent application No. 11/373,604entitled “SCALABLE VIDEO CODING WITH TWO LAYER ENCODING AND SINGLE LAYERDECODING” and owned by the assignee hereof, and which is incorporated byreference in its entirety herein.

Encoder First Pass Portion

FIG. 7 shows an illustrative example of the encoder 228 of FIG. 2. Theblocks shown illustrate various encoder processing that can be includedin encoder 228. In this example, the encoder 228 includes a first passportion 706 above a demarcation line 704, and a second pass portion 706(including functionality of second pass encoder 232 and re-encoder 234in FIG. 2) below the line 704.

The encoder 228 receives metadata and raw video from the preprocessor226. The metadata can include any metadata received or calculated by thepreprocessor 226, including metadata related to content information ofthe video. The first pass portion 702 of encoder 228 illustratesexemplary processes that can be included in first pass encoding 702,which is described below in terms of its functionality. As one of skillin the art will know, such functionality can be embodied in variousforms (e.g., hardware, software, firmware, or a combination thereof).

FIG. 7 illustrates an adaptive intra refresh (AIR) module. The AIRmodule 710 provides an input to an I-frame instantiation module 708which instantiates an I-frame based on the metadata. The first-passportion 702 can also include a content classification module 712configured to receive the metadata and video and determine contentinformation relating to the video. Content information can be providedto a rate control bit allocation module 714, which also receives themetadata and the video. The control bit allocation module 714 determinesrate bit control information and provides it to the mode decision module715. Content information and video can be provided to a intra-model(distortion) module 716, which provides intra-coding distortioninformation to the mode decision module 715 and a scalabilityrate-distortion for base and enhancement layer module 718. Video andmetadata are provided to a motion estimation (distortion) module 720which provides inter-coding distortion information to the scalabilityrate-distortion for base and enhancement layer module 718. Thescalability rate-distortion for base and enhancement layer module 718determines scalability rate-distortion information using distortionestimations from motion estimation module 720 and intra-model distortionmodule 716 which is provided to the mode decision module 715. The modedecision module 715 also receives input from the slice/MB orderingmodule 722. The slice/MB ordering module 722 receives input from anerror resilience module 740 (shown in the second pass portion 706), andprovides information on aligning independently encodable portions ofvideo (slices) with access unit boundaries for error resilience to themode decision module 715. The mode decision module 715 determinesencoding mode information based on its inputs and provides the “best”coding mode to the second pass portion 706. Further illustrativeexplanation of some examples of such first pass portion 702 encoding isdescribed below.

As stated above, the content classification module 712 receives themetadata and raw video supplied by the preprocessor 226. In someexamples, the preprocessor 226 calculates content information from themultimedia data and provides the content information to the contentclassification module 712 (e.g., in the metadata), which can use thecontent information to determine a content classification for themultimedia data. In some other aspects, the content classificationmodule 712 is configured to determine various content information fromthe multimedia data, and can also be configured to determine a contentclassification.

The content classification module 712 can be configured to determine adifferent content classification for video having different types ofcontent. The different content classification can result in differentparameters used in aspects of encoding the multimedia data, for example,determining a bit rate (e.g., bit allocation) for determiningquantization parameters, motion estimation, scalability, errorresiliency, maintaining optimal multimedia data quality across channels,and for fast channel switching schemes (e.g., forcing I-framesperiodically to allow fast channel switching. According to one example,the encoder 228 is configured to determine rate-distortion (R-D)optimization and bit rate allocations based on the contentclassification. Determining a content classification allows multimediadata to be compressed to a given quality level corresponding to adesired bit rate based on a content classification. Also, by classifyingthe content of the multimedia data (e.g., determining a contentclassification based on the Human Visual System), the resultingperceptive quality of communicated multimedia data on a display of areceiving device is made dependent on the video content.

As an example of a procedure that content classification module 712undergoes to classify content, FIG. 9 shows a process 900 illustratingan exemplary process by which the content classification module 712 mayoperate. As shown, the process 900 begins at input block 902 where thecontent classification module 712 receives of raw multimedia data andmetadata. The process 900 then proceeds to block 904 where the contentclassification module 712 determines spatial information and temporalinformation of the multimedia data. In some aspects, the spatial andtemporal information is determined by spatial and temporal maskings(e.g., filtering). The spatial and temporal information can bedetermined based on metadata that includes scene change data and motionvector (MV) smoothing. Process 900 then proceeds to block 912 whichperforms spatial complexity, temporal complexity, and sensitivityestimations. Process 900 then proceeds to block 916 where the content ofthe multimedia data is classified based on the results of the determinedspatial, temporal, and sensitivity data in blocks 904 and 912. Also inblock 916 a particular rate-distortion (R-D) curve can be selectedand/or R-D curve data can be updated. The process 900 then proceeds tooutput block 918, where the output can include a complexity-distortionmap or value indicating spatial and temporal activity (e.g., a contentclassification), and/or the selected R-D curves. Referring back to FIG.7, the content classification module 712 provides an output to a ratecontrol bit allocation module 714, an intra model (distortion) module716, and also to the I-Frame Instantiation module 708, discussed above.

Content Information

The content classification module 712 can be configured to calculate avariety of content information from the multimedia data, including avariety of content related metrics, including spatial complexity,temporal complexity, contrast ratio values, standard deviations andframe difference metrics, described further below.

The content classification module 712 can be configured to determinespatial complexity and temporal complexity of the multimedia data, andalso to associate a texture value to the spatial complexity and a motionvalue to the temporal complexity. The content classification module 712receives preprocessed content information relating to the contents ofthe multimedia data being encoded from the preprocessor 226, oralternatively, the preprocessor 226 can be configured to calculate thecontent information. As described above, the content information caninclude, for example, one or more D_(csat) values, contrast ratiovalues, motion vectors (MVs), and sum of absolute differences (SADs).

In general, multimedia data includes one or more sequences of images, orframes. Each frame can be broken up into blocks of pixels forprocessing. Spatial complexity is a broad term which generally describesa measure of the level of spatial details within a frame. Scenes withmainly plain or unchanging or low changing areas of luminance andchrominance will have low spatial complexity. The spatial complexity isassociated with the texture of the video data. Spatial complexity isbased on, in this aspect, a human visual sensitivity metric calledD_(csat), which is calculated for each block as a function of localspatial frequency and ambient lighting. Ordinary skilled artisans areaware of techniques for using spatial frequency patterns and lightingand contrast characteristics of visual images to take advantage of thehuman visual system. A number of sensitivity metrics are known fortaking advantage of the perspective limitations of the human visualsystem and could be used with method described herein.

Temporal complexity is a broad term which is used to generally describea measure of the level of motion in multimedia data as referencedbetween frames in a sequence of frames. Scenes (e.g., sequences offrames of video data) with little or no motion have a low temporalcomplexity. Temporal complexity can be calculated for each macroblock,and can be based on the D_(csat) value, motion vectors and the sum ofabsolute pixel differences between one frame and another frame (e.g., areference frame).

The frame difference metric gives a measure of the difference betweentwo consecutive frames taking into account the amount of motion(example, motion vector or MV) along with the residual energyrepresented as sum of absolute difference (SAD) between a predictor andthe current macroblock. Frame difference also provides a measure ofbidirectional or unidirectional prediction efficiencies.

One example of a frame difference metric based on the motion informationreceived from a pre-processor potentially performing motion compensatedde-interlacing is as follows. The deinterlacer performs a bidirectionalmotion estimation and thus bidirectional motion vector and SADinformation is available. A frame difference represented by SAD_MV foreach macroblock can be derived as follows:SAD _(—) MV=log₁₀ [SAD*exp(−min(1,MV))]  [29]where MV=Square_root(MV_(x) ²+MV_(y) ²), SAD=min(SAD_(N), SAD_(P)),where SAD_(N) is the SAD computed from the backward reference frame, andSAD_(P) is the SAD computed from the forward reference frame.

Another approach of estimating a frame difference was described above inreference to Equations 6-8. A SAD ratio (or contrast ration) γ can becalculated as earlier described above in Equation 6. A luminancehistogram of every frame can also be determined, the histogramdifference λ being calculated using Equation 7. The frame differencemetric D can be calculated as shown in Equation 8.

In one illustrative example, a contrast ratio and a frame differencemetric are utilized in the following manner to obtain a video contentclassification, which could reliably predict the features in a givenvideo sequence. Although described here as occurring in the encoder 228,a preprocessor 226 can also be configured to determine a contentclassification (or other content information) and pass the contentclassification to the encoder 228 via metadata. The process described inthe example below classifies the content into eight possible classes,similar to the classification obtained from the R-D curve basedanalysis. The classification process outputs a value in the rangebetween 0 and 1 for each superframe depending on the complexity of thescene and the number of scene change occurrences in that superframe. Thecontent classification module in the preprocessor can execute thefollowing steps (1)-(5) for each superframe to obtain a contentclassification metric from the frame contrast and frame differencevalues.

1. Calculate Mean Frame Contrast and Frame Contrast Deviation from themacroblock contrast values.

2. Normalize Frame Contrast and Frame Difference values using the valuesobtained from simulations, which are 40 and 5 respectively.

3. Compute a content classification metric using, e.g., the generalizedequation:

$\begin{matrix}{{CCMetric}\; = \;{{{CCW}\; 1*{I\_ Frame}\;{\_ Contrast}\;{\_ Mean}}\; + \;{{CCW}\; 2*{Frame\_ Difference}\;{\_ Mean}} - {{CCW}\; 3*{I\_ Contrast}\;{\;{\_ Deviation}^{\hat{}}}\; 2*\;{\exp\left( {{CCW}\; 4*{Frame\_ Difference}\;{\;{\_ Deviation}^{\hat{}}}\; 2} \right)}}}} & \lbrack 30\rbrack\end{matrix}$where CCW1, CCW2, CCW3 and CCW4 are weighting factors. In this example,the values are chosen to be 0.2 for CCW1, 0.9 for CCW2, 0.1 for CCW3 and−0.00009 for CCW4.

4. Determine the number of scene changes in the super frame. Generally,a super frame refers to a group of pictures or frames that can bedisplayed in a particular time period. Typically, the time period is onesecond. In some aspects, a super frame comprises 30 frames (for 30/fpsvideo). In other aspects a super frame comprises 24 frames (24/fpsvideo). Depending upon the number of scene changes, one of the followingcases gets executed.

-   -   (a) No Scene Changes: When there are no scene changes in a super        frame, the metric is entirely dependent only on the frame        difference values as shown in the following equation:        CCMetric=(CCW2+(CCW1/2))*Frame_Difference_Mean−(CCW3−(CCW1/2))*1*exp(−CCW4*Frame_Difference_Deviation^2)  [31]    -   (b) Single Scene Change: When there is a single scene change        frame observed in the superframe, the default equation would be        used to compute the metric, as shown below:

$\begin{matrix}{{CCMetric} = {{{CCW}\; 1*{I\_ Frame}{\_ Contrast}{\_ Mean}} + {{CCW}\; 2*{Frame\_ Difference}{\_ Mean}} - {{CCW}\; 3*{I\_ Contrast}{\_ Deviation}^{\bigwedge}2*{\exp\left( {{CCW}\; 4*{Frame\_ Difference}{\_ Deviation}^{\bigwedge}2} \right)}}}} & \lbrack 32\rbrack\end{matrix}$

-   -   (c) Two Scene Changes: When it is observed that there are at        most 2 scene changes in the given superframe, the last        superframe is accorded more weight than the first one as the        first one would be anyway refreshed by the latter quickly, as        shown in the following equation:

$\begin{matrix}{{CCMetric} = {{0.1*{I\_ Frame}{\_ Contrast}{\_ Mean}\; 1} + {\quad{{{CCW}\; 1*{I\_ Frame}{\_ Contrast}{\_ Mean}\; 2} + {\quad{\quad{{\left( {{{CCW}\; 2} - 0.1} \right)*{Frame\_ Difference}{\_ Mean}} - {{CCW}\; 3*{I\_ Contrast}{\_ Deviation1}^{\bigwedge} 2*{\quad{{I\_ Contrast}{\_ Deviation}\; 2^{\bigwedge} 2*{\exp\left( {{CCW}\; 4*{Frame\_ Difference}{\_ Deviation}^{\bigwedge}2} \right)}{\quad\quad}}}}}}}}}}} & \lbrack 33\rbrack\end{matrix}$

-   -   (d) Three or more Scene Changes: If the given superframe is        observed to have more than 3 I frames (say N), the last I frame        is given more weight and all other I frames are given a weight        of 0.05, as shown in the following equation:

$\begin{matrix}{{CCMetric} = {{0.05*{I\_ Frame}{\_ Contrast}{\_ Mean}_{({{1\mspace{11mu}\ldots\mspace{11mu} N} - 1})}} + {{CCW}\; 1*{I\_ Frame}{\_ Contrast}{\_ Mean}_{(N)}} + {\left( {{{CCW}\; 2} - \left( {0.05*\left( {N - 1} \right)} \right)} \right)*{Frame\_ Difference}{\_ Mean}} - {{CCW}\; 3*{I\_ Contrast}{{\_ Deviation}_{(N)}}^{\bigwedge}2*{I\_ Contrast}{{\_ Deviation}_{({{1\mspace{11mu}\ldots\mspace{11mu} N} - 1})}}^{\bigwedge} 2*{\exp\left( {{CCW}\; 4*{Frame\_ Difference}{\_ Deviation}^{\bigwedge}2} \right)}}}} & \lbrack 34\rbrack\end{matrix}$

5. A correction is may be used for the metric in the case of low motionscenes when the Frame Difference mean is less than 0.05. An offset of(CCOFFSET) 0.33 would be added to the CCMetric.

The content classification module 712 uses the D_(csat) value, motionvectors and/or the sum of absolute differences to determine a valueindicating a spatial complexity for the macroblock (or designated amountof video data). The temporal complexity is determined by a measure ofthe frame difference metric (the difference between two consecutiveframes taking into account the amount of motion, with motion vectors,and the sum of absolute differences between the frames).

In some aspects, the content classification module 712 can be configuredto generate a bandwidth map. For example, bandwidth map generation canbe performed by the content classification module 712 if thepreprocessor 226 does not generate a bandwidth map.

Determining Texture and Motion Values

For each macroblock in the multimedia data, the content classificationmodule 712 associates a texture value with the spatial complexity and amotion value with the temporal complexity. The texture value relates tothe luminescence values of the multimedia data, where a low texturevalue indicates small changes in luminescence values of neighboringpixels of the data, and a high texture value indicates large changes inthe luminescence values of neighboring pixels of the data. Once thetexture and motion values are calculated, the content classificationmodule 712 determines a content classification by considering both themotion and texture information. The content classification module 712associates the texture for the video data being classified with arelative texture value, for example, “Low” texture, “Medium” texture, or“High” texture, which generally indicates the complexity of luminancevalues of the macroblocks. Also, the content classification module 712associates the motion value calculated for the video data beingclassified with a relative motion value, for example, “Low” motion,“Medium” motion, or “High” motion which generally indicates the amountof motion of the macroblocks. In alternative aspects, fewer or morecategories for motion and texture can be used. Then, a contentclassification metric is then determined by considering the associatedtexture and motion values.

FIG. 8 illustrates an example of a classification chart that illustrateshow texture and motion values are associated with an contentclassification. A person of ordinary skill in the art is familiar withmany ways to implement such a classification chart, for example, in alook-up table or a database. The classification chart is generated basedon predetermined evaluations of video data content. To determine thevideo data classification, a texture value of “Low,” “Medium,” or “High”(on the “x-axis”) is cross-referenced with a motion value of “Low,”“Medium,” or “High” (on the “y-axis”). A content classificationindicated in the intersecting block is assigned to the video data. Forexample, a texture value of “High” and a motion value of “Medium”results in a classification of seven (7). FIG. 8 illustrates variouscombinations of relative texture and motion values that are associatedwith eight different content classifications, in this example. In someother aspects, more or fewer classifications can be used. Furtherdescription of an illustrative aspect of content classification isdisclosed in co-pending U.S. patent application Ser. No. 11/373,577entitled “CONTENT CLASSIFICATION FOR MULTIMEDIA PROCESSING” filed onMar. 10, 2006, assigned to the assignee hereof and hereby expresslyincorporated by reference herein.

Rate Control Bit Allocation

As described herein, a multimedia data content classification can beused in encoding algorithms to effectively improve the bit managementwhile maintaining a constant the perceptive quality of video. Forexample, the classification metric can be used in algorithms forscene-change detection, encoding bit rate allocation control, and framerate up conversion (FRUC). Compressor/decompressor (codec) systems anddigital signal processing algorithms are commonly used in video datacommunications, and can be configured to conserve bandwidth, but thereis a trade-off between quality and bandwidth conservation. The bestcodecs provide the most bandwidth conservation while producing the leastdegradation of video quality.

In one illustrative example, the rate control bit allocation module 714uses the content classification to determine a bit rate (e.g., thenumber of bits allocated for encoding the multimedia data) and storesthe bit rate into memory for use by other process and components of theencoder 228. A bit rate determined from the classification of the videodata can help conserve bandwidth while providing multimedia data at aconsistent quality level. In one aspect, a different bit rate can beassociated with each of the eight different content classifications andthen that bit rate is used to encode the multimedia data. The resultingeffect is that although the different content classifications ofmultimedia data are allocated a different number of bits for encoding,the perceived quality is similar or consistent when viewed on a display.

Generally, multimedia data with a higher content classification areindicative of a higher level of motion and/or texture and is allocatedmore bits when encoded. Multimedia data with a lower classification(indicative of less texture and motion) is allocated less bits. Formultimedia data of a particular content classification, the bit rate canbe determined based on a selected target perceived quality level forviewing the multimedia data. Determining multimedia data quality can bedetermined by humans viewing and grading the multimedia data. In somealternative aspects, estimates of the multimedia data quality can bemade by automatic test systems using, for example, signal to noise ratioalgorithms. In one aspect, a set of standard quality levels (e.g., five)and a corresponding bit rate needed to achieve each particular qualitylevel are predetermined for multimedia data of each contentclassification. To determine a set of quality levels, multimedia data ofa particular content classification can be evaluated by generating aMean Opinion Score (MOS) that provides a numerical indication of avisually perceived quality of the multimedia data when it is encodedusing a certain bit rate. The MOS can be expressed as a single number inthe range 1 to 5, where 1 is lowest perceived quality, and 5 is thehighest perceived quality. In other aspects, the MOS can have more thanfive or fewer than five quality levels, and different descriptions ofeach quality level can be used.

Determining multimedia data quality can be determined by humans viewingand grading the multimedia data. In some alternative aspects, estimatesof the multimedia data quality can be made by automatic test systemsusing, for example, signal to noise ratio algorithms. In one aspect, aset of standard quality levels (e.g., five) and a corresponding bit rateneeded to achieve each particular quality level are predetermined formultimedia data of each content classification.

Knowing the relationship between the visually perceived quality leveland a bit rate for multimedia data of a certain content classificationcan be determined by selecting a target (e.g., desired) quality level.The target quality level used to determine the bit rate can bepreselected, selected by a user, selected through an automatic processor a semi-automatic process requiring an input from a user or fromanother process, or be selected dynamically by the encoding device orsystem based on predetermined criteria. A target quality level can beselected based on, for example, the type of encoding application, or thetype of client device that will be receiving the multimedia data.

In the illustrated example in FIG. 7, the rate control bit allocationmodule 714 receives both data from the content classify classificationmodule 712 and metadata directly from the preprocessor 226. The ratecontrol bit allocation module 714 resides in the first pass portion ofthe encoder 228, and a rate control fine tuning module 738 resides inthe second pass portion 706. This two-pass rate control aspect isconfigured such that the first-pass (rate control bit allocation module714) performs context adaptive bit allocation with one superframelook-ahead (e.g., targeting long term average bit rates of 256 kbps) andlimits the peak rate, and the second-pass (rate control fine tuningmodule 738) refines the first-pass results for two-layer scalability andperforms rate adaptation. The rate control operates on four levels: (1)GOP level—controls bit distribution of I, P, B, and F frames to benon-uniform inside a GOP; (2) super frame level—controls hard limits onmaximum super frame size; (3) frame level—controls bit requirementsaccording to the spatial and temporal complexity of the multimedia dataframes, which are based on the content information (e.g., a contentclassification); and (4) macroblock level—controls bit allocation ofmacroblocks based on spatial and temporal complexity maps, which arebased on the content information (e.g., a content classification).

An exemplary flow diagram of the operation of the rate control module714 is illustrated in FIG. 10. As shown in FIG. 10, the process 1000begins at an input 1002 block. The rate control module 714 receivesvarious inputs, not all of which are necessarily illustrated by FIG. 7.For example, input information can include metadata from thepreprocessor 226 a target bitrate, encoder buffer size (or, as anequivalent, the maximum delay time for rate control), an initial ratecontrol delay, and frame rate information. Further input information caninclude inputs at the group of pictures (GOP) level, including, forexample, maximum super frame size, length and P/B-frame distribution ofthe GOP (including scene change information), arrangement of base andenhancement layers desired, a complexity-distortion metric for picturesin the GOP for a future 30 frames. Other input information includesinputs at the picture level, including complexity-distortion map for thecurrent picture (received from the content classification module 712),quantization parameters (QP), and bit breakdown of past 30 frames (fitover a sliding window). Finally, input information at the macroblock(MB) level includes, for example, the mean absolute difference (MAD) ofcollocated macroblocks (MB) in a reference picture, and a coded blockpattern (CBP) of macroblocks after quantization (whether skipped ornot).

After the inputs at block 1002, process 1000 proceeds to block 1004 forinitialization for encoding the bitstream. Concurrently, a bufferinitialization 1006 is performed. Next, a GOP is initialized as shown inblock 1008, with GOP bit allocation 1010 received as part of theinitialization. After GOP initialization, flow proceeds to block 1012,where a slice is initialized. This initialization includes an update ofthe header bits as shown by block 1014. After the initializations ofblock 1004, 1008 and 1012 are performed, rate control (RC) for a basicunit or macroblock (MB) is carried out as shown by block 1016. As partof the rate control determination of a macroblock in block 1016, inputsare received via interfaces in the encoder 228. These inputs can includemacroblock (MB) bit allocation 1018, an update of quadratic modelparameters 1020, and an update of median absolute deviation from themedian (“MAD,” a robust estimate of dispersion) parameters 1022. Nextprocess 1000 proceeds to block 1024 for execution of operations afterencoding one picture 1024. This procedure includes receiving an updateof buffer parameters as shown by block 1026. Process 1000 then proceedsto output block 1028 where the rate control module 714 outputsquantization parameters QP for each macroblock MB to be used by a modedecision module 715 as shown in FIG. 7

Motion Estimation

Motion estimation module 720 receives inputs of metadata and raw videofrom the preprocessor 226, and provides output that can include blocksize, motion vectors distortion metrics, and reference frame identifiersto a mode decision module 715. FIG. 11 illustrates an exemplaryoperation of the motion estimation module 720. As shown, process 1100begins with an input 1102. At the frame level, the module 720 receivesinput of the reference frame ID and motion vectors. At the macroblocklevel, input 1102 includes input pixels and reference frame pixels.Process 1100 continue to step 1104 wherein color motion estimation (ME)and motion vector prediction are performed. In order to carry out thisprocess, various inputs are received including MPEG-2 motion vectors,and luma motion vectors MVs 1106, motion vector smoothing 1108, andnon-causal motion vectors 1110. Next, process 1100 proceeds to block1112 where a motion vector search algorithm or methodology is performed,such as hexagonal or diamond search methods. Inputs to the process atblock 1112 may include sum of absolute difference (SAD), sum of squareddifference (SSD), and/or other metrics as shown by block 1114. Once amotion vector search is performed, process 1100 proceeds to terminationblock 1116, where termination processing is performed. The process 100then ends at output block 1118, which yields an output of block size,motion vector (MV), distortion metrics, and Reference Frame identifiers.

Scalability R-D for Base and Enhancement Layer

FIG. 13 illustrates an exemplary flow diagram of a scalability process1300 that can be performed by the scalability R-D module 718. Process1300 begins at start block 1302 and proceeds to block 1304 where thescalability R-D module 718 it receives an input from motion estimationmodule 720 and performs motion estimation. Motion estimation relies oninput of base layer reference frames, enhancement layer referenceframes, and the to-be-coded original frame as indicated by block 1306.Such information can be calculated by the GOP partitioner 612 andcommunicated to the scalability R-D module 718 via, e.g., metadata. Theprocess 1300 proceeds to block 1308 to determine scalability informationof the data base layer and enhancement layer data. Base layer encodingis next performed as shown in block 1310, followed by enhancement layerencoding in block 1312. The encoding of the enhancement layer can usethe base layer coding results for interlayer prediction as in input, asillustrated by block 1314, thus temporally it is performed after baselayer encoding. This is further described in co-pending U.S. patentapplication No.11/373,604 entitled “SCALABLE VIDEO CODING WITH TWO LAYERENCODING AND SINGLE LAYER DECODING.” After encoding is complete, process1300 ends at block 1316.

Slice/Macroblock Ordering

The first pass portion 702 also includes a slice/macroblock orderingmodule 722, which receives an input from an error resilience module 740in the second pass portion and provides an slice alignment informationto the mode decision module 715. Slices are chunks of independentlydecodable (entropy decoding) coded video data. Access units (AU) arecoded video frames each comprising a set of NAL units always containingexactly one primary coded picture. In addition to the primary codedpicture, an access unit may also contain one or more redundant codedpictures or other NAL units not containing slices or slice datapartitions of a coded picture. The decoding of an access unit alwaysresults in a decoded picture.

Frames can be time division multiplexed blocks of physical layer packets(called a TDM capsule) that offer the highest time diversity. Asuperframe corresponds to one unit of time (e.g., 1 sec) and containsfour frames. Aligning slice and AU boundaries to frame boundaries in thetime domain results in the most efficient separation and localization ofcorrupted data. During a deep fade, most of the contiguous data in a TDMcapsule is affected by errors. Due to time diversity, the remaining TDMcapsules have a high probability of being intact. The uncorrupted datacan be utilized to recover and conceal the lost data from the affectedTDM capsule. Similar logic applies to frequency domain multiplexing(FDM) where frequency diversity is attained through separation infrequency subcarriers that the data symbols modulate. Furthermore,similar logic applies to spatial (through separation in transmitters andreceivers antennas) and other forms of diversity often applied inwireless networks.

In order to align slices and AU to frames, the outer code (FEC) codeblock creation and MAC layer encapsulation should align as well. FIG. 20illustrates the organization of coded video data or a video bitstream inslices and AUs. The coded video may be constituted in one or morebitstreams, e.g., base layer bitstream and enhancement layer bitstreamwhere layered video coding is applied.

The video bitstream comprises AUs as illustrated in FIG. 20 by Frame 1′2005, Frame 3′ 2010 and Frame M′ 2015. The AUs comprise of slices ofdata, as illustrated by Slice 1 2020, slice 2 2025, and slice N 2030.Each start of slice is identified by a start code and provides tonetwork adaptation. In general, I-frame or intra coded AUs are large,followed by P-frames or forward predicted frames followed by B-frames.Coding an AU into multiple slices incurs a significant overhead cost interms of the coded bitrate because spatial prediction across slices isrestricted and slice headers contribute to the overhead too. Becauseslice boundaries are resynchronization points, restricting contiguousphysical layer packets to slices controls errors since when a PLP iscorrupted, error is confined to the slice in the PLP whereas if the PLPcontained multiple slices or parts of multiple slices, the error wouldimpact all slices or portions of slices in the PLP.

Since I-frames are typically large, for example, on the order of 10's ofkbits, the overhead due to multiple slices is not a large proportion ofthe total I-frame size or total bitrate. Also, having more slices in anintra-coded AU enables better and more frequent resynchronization andmore efficient spatial error concealment. Also, I-frames carry the mostimportant information in the video bitstream since P and B-frames arepredicted off of I-frames. I-frames also serve as random access pointsfor channel acquisition.

Referring now to FIG. 21, carefully aligning the I-frames to frameboundaries and the slices with an I AU to frame boundaries as well,enables the most efficient error control, error protection (since if oneslice that belonged to Frame 1 2105 is lost, slices that belong to Frame2 2110 have a high probability of being intact because Frame 2 2110 hasa significant time separation from Frame 1 2105 error recovery can beperformed through resynchronization and error concealment.

Because P-frames are typically sized on the order of a few kbits,aligning slices of a P-frame and integer number of P-frames to frameboundaries enables error resilience without a detrimental loss ofefficiency (for similar reasons as those for I-frames). Temporal errorconcealment can be employed in such aspects. Alternatively, dispersingconsecutive P-frames such that they arrive in different frames providesadded time diversity among P-frames, which can be because temporalconcealment is based on motion vectors and data from previouslyreconstructed I or P frames. B-frames can be extremely small (100's ofbits) to moderately large (few 1000 bits). Hence aligning integer numberof B-frames to frame boundaries is desirable to achieve error resiliencywithout a detrimental loss of efficiency.

Mode Decision Module

FIG. 12 illustrates some examples of the operation of the mode decisionmodule 715. As shown, the process 1200 begins at an input block 1202. Inone illustrative example, the various information input to the modedecision module 715 include slice type, Intra 4×4cost, Intra 16×16cost,IntraUV 8×8cost, IntraY 16×16 Mode, IntraUV Mode, motion vector data(MVD), quantization parameters (QP), SpPredMB4×4Y, SpPredMB16×16Y,SpPredMB8×8U, SpPredMB8×8V, Rate Distortion Flag, Raw YMB pixels, RawUMB Pixels, and Raw VMB Pixels. Process 1200 then proceeds to block 1204encoding initialization, which can be initiated by an input signal orinterface directing encoder initialization as indicated by block 1206.The initialization can include setting allowed modes (including skip,direct), set mode weights (if required, the default will be equalweights for all modes), and setting buffers. After initialization,process 1200 proceeds to block 1208 where main processing for modedecision is performed, including: computing macroblock (MB) mode costsfor each allowed mode, weighting of each MB mode cost with a weightingfactor, and selecting a minimum MB mode cost mode. Inputs involved withthese operations include motion estimation (e.g., MVD and predictions)and spatial prediction (e.g., all intra costs and predictions) asillustrated by blocks 1210 and 1212. Interfaced with the mode decisionmodule 715 is entropy encoding in block 1214 that, among other things,improves the compression rate. Process 1200 the proceeds to block 1216where buffers are updated to pass information to the encoder second passportion 706. Finally, process 1200 proceeds to block 1218 where the“best” encoding mode can be communicated to the encoder second passportion 706.

Encoder Second Pass Portion

Referring again to FIG. 7, the second pass portion 706 of the encoder228 includes a second pass encoder module 232 for performing the secondpass of encoding. The second pass encoder 232 receives the output fromthe mode decision module 715. The second pass encoder 232 includes aMC/Transform Quantization module 726 and a Zig Zag (ZZ)/Entropy encoder728. The results of the second pass encoder 232 are output to ascalability module 730 and a bitstream packing module 731, which outputsa encoded base and enhancement layer for transmission by the transcoder200 via a synchronizing layer 240 (illustrated in FIG. 2). As shown inFIG. 2, it is noted that base and enhancement layers from the secondpass encoder 232 and the re-encoder 234 are assembled by thesynchronizing layer 240 into a packetized PES 242 including base andenhanced layers, a data PES 244 (e.g., CC and other text data, and anaudio PES 246. It is noted that the audio encoder 236 receives decodedaudio information 218 and, in turn, encodes the information and outputsthe encoded information 238 to the synch layer 240.

Re-encoder

Referring again to FIG. 7, the encoder second pass portion 706 alsoincludes a re-encoder 234, which corresponds to re-encoder 234 in FIG.2. The re-encoder 234 also receives the output of first pass portion 702and includes a MC/Transform Quantization 726 and ZZ/Entropy coding 728portions. Additionally, the scalability module 730 outputs to there-encoder 234. The re-encoder 234 outputs the resultant base andenhanced layer from re-encoding to the bitstream packing module 731 fortransmission to a synchronizer (e.g., sync. layer 240 shown in FIG. 2).The encoder 228 example in FIG. 7 also includes a rate control finetuning module 738 which provides bitstream packing feedback to both theMC/transform quantization module 234 in the second pass encoder 232 andto the ZZ/Entropy module 736 in the re-encoder 234 to help tune thesecond pass encoding (e.g., to increase compression efficiency).

Error Resilience Module

The encoder 228 example illustrated in FIG. 7 also includes an errorresilience module 740 in the second pass portion 706. The errorresilience module 740 communicates with the bitstream packing module 731and the slice/MB ordering module 722. The error resilience module 740receives metadata from the preprocessor 228 and selects an errorresilience scheme, for example, aligning slice and access units to frameboundaries, predictive hierarchy and adoptive intra refresh. Theselection of the error resilience scheme can be based on informationreceived in the metadata, or from information communicated to the errorresilience module from the bitstream packing module 731 and the slice/MBordering module 722. The error resilience module 740 providesinformation to the slice/macroblock (MB) ordering module in first passportion 702 to implement the selected error resiliency processes. Videotransmissions over error prone environments can employ error resiliencestrategies and algorithms that can result in presenting clearer and lesserror-filled data to a viewing user. The error resiliency descriptionbelow can apply to any individual or combination of existing or futureapplication, transport and physical layer or other technologies.Effective error robustness algorithms integrates an understanding oferror susceptibility properties and error protection capabilities amongthe OSI layers in conjunction with desirable properties of thecommunication system such as low latency and high throughput. Errorresiliency processing can be based on the content information of themultimedia data, for example, on the content classification of themultimedia data. One of the primary advantages is recoverability fromfading and multi-path channel errors. The error resilience approachesdescribed below pertain specifically to processes that can beincorporated in the encoder 228 (e.g., in particular in error resiliencemodule 740 and slice/MB ordering module 722), and can be extendedgenerically to data communication in error prone environments.

Error resilience

For a prediction based hybrid compression system, intra-coded frames areindependently coded without any temporal prediction. Inter-coded framescan be temporally predicted from past frames (P-frames) and futureframes (B-frames). The best predictor can be identified through a searchprocess in the reference frame (one or more) and a distortion measuresuch as SAD is used to identify the best match. The predictive codedregion of the current frame can be a block of varying size and shape(16×16, 32×32, 8×4 etc) or a group of pixels identified as an objectthrough, for example, segmentation.

Temporal prediction typically extends over many frames (e.g., 10 to 10'sof frames) and is terminated when a frame is coded as an I-frame, theGOP typically being defined by the I-frame frequency. For maximum codingefficiency, a GOP is a scene, for example, GOP boundaries are alignedwith scene boundaries and scene change frames are coded as I-frames. Inlow motion sequences comprise a relatively static background and motionis generally restricted to the foreground object. Examples of content ofsuch low motion sequences include news and weather forecasts programswhere more than 30% of most viewed content is of this nature. In lowmotion sequences, most of the regions are inter-coded and the predictedframes refer back to the I-frame through intermediate predicted frames.

Referring to FIG. 22, the intra-coded block 2205 in the I-frame is thepredictor for the inter-coded block 2210 in coded frame (or AU) P1. Inthis example, the region of these blocks is a stationary part of thebackground. Through consecutive temporal prediction, the sensitivity ofthe intra-coded block 2205 to errors goes up since it is a “good”predictor which also implies that its “importance” is higher.Additionally, the intra-coded block 2205, by virtue of this chain oftemporal prediction called the prediction chain, persists longer in thedisplay (for the duration of the scene in the example in the figure).

Prediction hierarchy is defined as the tree of blocks created based onthis importance level or measure of persistence with the parent at thetop (intra coded block 2205) and the children at the bottom. Note thatthe inter coded block 2215 in P1 is on the second level of the hierarchyand so on. Leaves are blocks that terminate a prediction chain.

Prediction hierarchy can be created for video sequences irrespective ofcontent type (such as music and sports as well and not just news) and isapplicable to prediction based video (and data) compression in general(this applies to all the inventions described in this application). Oncethe prediction hierarchy is established, error resilience algorithmssuch as adaptive intra refresh, described below, can be applied moreeffectively. The importance measure can be based on recoverability of agiven block from errors such as through concealment operations andapplying adaptive intra refresh to enhance resilience of the codedbitstream to errors. An estimate of the importance measure can be basedon number of times a block is used as a predictor also referred to asthe persistence metric. The persistence metric is also used to improvecoding efficiency by arresting prediction error propagation. Thepersistence metric also increases bit allocation for the blocks withhigher importance.

Adaptive Intra Refresh

Adaptive intra refresh is an error resilience technique that can bebased on content information of the multimedia data. In an intra refreshprocess, some MBs are intra-coded even though standard R-D optimizationwould dictate that they should be inter-coded MBs. AIR employsmotion-weighted intra refresh to introduce intra-coded MBs in P orB-frames. These intra-coded MBs, contained in the base layer, can beencoded with either QP_(b) or QP_(e). If QP_(e), is used for the baselayer, then no refinement should be done at the enhancement layer. IfQP_(b), is used for the base layer, then refinement may be appropriate,otherwise at the enhancement layer, the drop of quality will benoticeable. Since inter-coding is more efficient than intra-coding inthe sense of coding efficiency, these refinements at the enhancementlayer will be inter-coded. This way, the base layer coefficients willnot be used for the enhancement layer, and the quality is improved atthe enhancement layer without introducing new operations.

In some aspects, adaptive intra refresh can be based on contentinformation of the multimedia data (e.g., a content classification)instead of, or in addition to, a motion weighted basis. For example, ifthe content classification is relatively high (e.g., scenes having highspatial and temporal complexity) adaptive intra refresh can introducerelatively more intra-coded MB's into P or B-frames. Alternatively, ifthe content classification was relatively low (indicating a less dynamicscene with low spatial and/or temporal complexity) Adaptive intrarefresh can introduce fewer intra coded MB's in the P and B-frames. Suchmetrics and methods for improving error resiliency can be applied notjust in the context of wireless multimedia communications but towardsdata compression and multimedia processing in general (e.g., in graphicsrendering).

Channel Switch Frame

A channel switch frame (CSF) as defined herein is a broad termdescribing a random access frame inserted at an appropriate location ina broadcast stream for fast channel acquisition and thus fast channelchange between streams in a broadcast multiplex. Channel switch framesalso increase error robustness, as they provide redundant data that canbe used if the primary frame is transmitted with an error. An I-frame ora progressive I-frame such as the progressive decoder refresh frame inH.264 is typically serves as a random access point. However, frequentI-frames (or short GOPs, shorter than scene durations) results in asignificant reduction in compression efficiency. Because intra-codedblocks may be used for error resilience, random access and errorresilience can be effectively combined through prediction hierarchy toimprove coding efficiency while increasing robustness to errors.

Improvement of random access switching and error robustness can beachieved in concert, and can be based on content information such as acontent classification. For low motion sequences, prediction chains arelong and a significant portion of the information required toreconstruct a superframe or scene is contained in the I-frame thatoccurred at the start of the scene. Channel errors tend to be bursty andwhen a fade strikes and FEC and channel coding fail, there is heavyresidual error that concealment fails. This is particularly severe forlow motion (and hence low bit rate) sequences since the amount of codeddata is not significant enough to provide good time diversity within thevideo bitstream and because these are highly compressible sequences thatrenders every bit useful for reconstruction. High motion sequences aremore robust to errors due to the nature of content—more new informationin every frame increases the number of coded intra blocks which areindependently decodable and more resilient to error inherently. Adaptiveintra refresh based on prediction hierarchy achieves a high performancefor high motion sequences and performance improvement is not significantfor low motion sequences. Hence, a channel switch frame containing mostof the I-frame is a good source of diversity for low motion sequences.When an error strikes a superframe, decoding in the consecutive framestarts from the CSF which recovers the lost information due toprediction and error resilience is achieved.

In the case of high motion sequences such as sequences having arelatively high content classification (e.g., 6-8), the CSF can consistsof blocks that persist in the SF—those that are good predictors. Allother regions of the CSF do not have to be coded since these are blocksthat have short prediction chains which implies that they are terminatedwith intra blocks. Hence, CSF still serves to recover from lostinformation due to prediction when an error occurs. CSFs for low motionsequences are on par with the size of I-frames, and they can be coded ata lower bit rate through heavier quantization, where CSFs for highmotion sequences are much smaller than the corresponding I-frames.

Error resilience based on prediction hierarchy can work well withscalability and can achieve a highly efficient layered coding.Scalability to support hierarchical modulation in physical layertechnologies may require data partitioning of the video bitstream withspecific bandwidth ratios. These may not always be the ideal ratios foroptimal scalability (for example, with the least overhead). In someaspects, a 2-layer scalability with 1:1 bandwidth ratio is used.Partitioning video bitstream to 2-layers of equal size may not be asefficient for low motion sequences. For low motion sequences, the baselayer containing all header and metadata information is larger than theenhancement layer. However, since CSFs for low motion sequences arelarger, they fit nicely in the remaining bandwidth in the enhancementlayer.

High motion sequences have sufficient residual information that datapartitioning to 1:1 can be achieved with the least overhead.Additionally, a channel switch frame for such sequences are much smallerfor high motion sequences. Hence, error resilience based on predictionhierarchy can work well with scalability for high motion sequences aswell. Extending the concepts discussed above for moderate motion clipsis possible based on the descriptions of these algorithms, and theproposed concepts apply for video coding in general.

Multiplexer

In some encoder aspects, a multiplexer can be used for encoding multiplemultimedia streams produced by the encoder and used to prepare encodedbits for broadcast. For example, in the illustrative aspect of encoder228 shown I FIG. 2, the synch layer 240 comprises a multiplexer. Themultiplexer may be implemented to provide the bit rate allocationcontrol. The estimated complexity can be provided to the multiplexer,which can then allocate the available bandwidth for a collection ofmultiplexed video channels according to the encoding complexityanticipated for those video channels, which then permits the quality ofa particular channel to remain relatively constant even if the bandwidthfor the collection of multiplexed video streams is relatively constant.This provides a channel within a collection of channels to have avariable bit rate and relatively constant visual quality, rather than arelatively constant bit rate and a variable visual quality.

FIG. 18 is a block diagram illustrating a system of encoding multiplemultimedia streams or channels 1802. The multimedia streams 1802 areencoded by respective encoders 1804, which are in communication when amultiplexer (MUX) 1806, which in turn is in communication with atransmission medium 1808. For example, the multimedia streams 1802 cancorrespond to various content channels, such as news channels, sportschannels, movie channels, and the like. The encoders 1804 encode themultimedia streams 1802 to the encoding format specified for the system.While described in the context of encoding of video streams, theprinciples and advantages of the disclosed techniques are generallyapplicable to multimedia streams including, for example, audio streams.The encoded multimedia streams are provided to a multiplexer 1806, whichcombines the various encoded multimedia streams and sends the combinedstream to the transmission medium 1808 for transmission.

The transmission medium 1808 can correspond to a variety of mediums,such as, but not limited to, digital satellite communication, such asDirecTV®, digital cable, wired and wireless Internet communications,optical networks, cell phone networks, and the like. The transmissionmedium 1808 can include, for example, modulation to radio frequency(RF). Typically, due to spectral constraints and the like, thetransmission medium has a limited bandwidth and the data from themultiplexer 1806 to the transmission medium is maintained at arelatively constant bit rate (CBR).

In conventional systems, the use of constant bit rate (CBR) at theoutput of the multiplexer 1806 may require that the encoded multimediaor video streams that are input to the multiplexer 1806 are also CBR. Asdescribed in the background, the use of CBR when encoding video contentcan result in a variable visual quality, which is typically undesirable.

In the illustrated system, two or more of the encoders 1804 communicatean anticipated encoding complexity of input data. One or more of theencoders 1804 may receive adapted bit rate control from the multiplexer1806 in response. This permits an encoder 1804 that expects to encoderelatively complex video to receive a higher bit rate or higherbandwidth (more bits per frame) for those frames of video in aquasi-variable bit rate manner. This permits the multimedia stream 1802to be encoded with a consistent visual quality. The extra bandwidth thatis used by a particular encoder 1804 encoding relatively complex videocomes from the bits that would otherwise have been used for encodingother video streams 1804 if the encoders were implemented to operate atconstant bit rates. This maintains the output of the multiplexer 1806 atthe constant bit rate (CBR).

While an individual multimedia stream 1802 can be relatively “bursty,”that is, vary in used bandwidth, the cumulative sum of multiple videostreams can be less bursty. The bit rate from channels that are encodingless complex video that can be reallocated by, for example, themultiplexer 1806, to channels that are encoding relatively complexvideo, and this can enhance the visual quality of the combined videostreams as whole.

The encoders 1804 provide the multiplexer 1806 with an indication of thecomplexity of a set of video frames to be encoded and multiplexedtogether. The output of the multiplexer 1806 should provide an outputthat is no higher than the bit rate specified for the transmissionmedium 1808. The indications of the complexity can be based on thecontent classification as discussed above to provide a selected level ofquality. The multiplexer 1006 analyzes the indications of complexity,and provides the various encoders 1004 with an allocated number of bitsor bandwidth, and the encoders 1804 use this information to encode thevideo frames in the set. This permits a set of video frames toindividually be variable bit rate, and yet achieve constant bit rate asa group.

Content Classification can also be used in enabling quality basedcompression of multimedia in general for any generic compressor. ContentClassification and the methods and apparatuses described here may beused in quality based and/or content based multimedia processing of anymultimedia data. One example is its use in compression of multimedia ingeneral for any generic compressor. Another example is in decompressionor decoding in any decompressor or decoder or post-processor such asinterpolation, resampling, enhancement, restoration and presentationoperations.

Referring now to FIG. 19, a typical video communication system includesa video compression system consisting of a video encoder and a videodecoder connected by a communication network. Wireless networks are oneclass of error prone networks where the communication channel exhibitslog-normal fading or shadowing and multi-path fading in mobile scenariosin addition to path loss. In order to combat channel errors and providea reliable communications for application layer data, the RF modulatorincludes forward error correction including interleavers and channelcoding such as convolutional or turbo coding.

Video compression reduces redundancy in the source video and increasesthe amount of information carried in each bit of the coded video data.This increases the impact in quality when even a small portion of thecoded video is lost. Spatial and temporal prediction inherent in videocompression systems aggravates the loss and causes errors to propagateresulting in visible artifacts in the reconstructed video. Errorresilience algorithms at the video encoder and error recovery algorithmsat the video decoder enhance the error robustness of the videocompression system.

Typically, the video compression system is agnostic to the underlyingnetwork. However, in error prone networks, integrating or aligning errorprotection algorithms in the application layer with FEC and channelcoding in the link/physical layers is highly desirable and provides themost efficiency in enhancing error performance of the system.

FIG. 14 illustrates one example of a rate-distortion data flow that canoccur in the encoder 228 to encode frames. Process 1400 begins at start1402, and proceeds to decision block 1404, where it receives scenechange detector input 1410 from the preprocessor 226 (e.g., viametadata) and error resilience input 1406 is acquired. If theinformation indicates a selected frame is an I frame, the processintra-encodes the frame. If the information indicates the selected frameis a P or B frame, the process uses intra-coding and motion estimation(inter) coding to encode the frame.

After an affirmative condition occurs for the conditions of block 1404,process 1400 proceeds to a preparation block 1414 where the rate R isset to value R=Rqual, the desired target quality based on R-D curves.This setting is received from a data block 1416 comprising R-D curves.Process 1400 then proceeds from to block 1418 where Rate Control BitAllocation {Qpi} is performed based on image/video activity information(e.g., a content classification) from a content classification processat block 1420.

The rate control bit allocation block 1418 is used, in turn, for motionestimation in block 1422. The motion estimation 1422 can also receiveinput of metadata from the preprocessor 1412, motion vector smoothing(MPEG−2+History) from block 1424 and multiple reference frames(causal+non-causal macroblock MBs) from block 1426. Process 1400 thenproceeds to block 1428 where rate calculations for intra-coded modes aredetermined for the rate control bit allocation (Qpi). Process 1400 nextproceeds to block 1430 where mode and quantization parameters aredetermined. The mode decision of block 1430 is made based on the motionestimation of block 1422 input, error resilience 1406 input, andscalability R-D, which is determined at block 1431. Once the mode isdecided, flow proceeds to block 1432. It is noted that the flow fromblock 1430 to 1432 occurs when data is passed from the first pass to thesecond pass portions of the encoder.

At block 1432, transform and quantization is performed by the secondpass of the encoder 228. The transform/quantization process is adjustedor fine tuned as indicated with block 1444. This transform/quantizationprocess may be influenced by a rate control fine tuning module (FIG. 7).Process 1400 then proceeds to block 1434 for zigzag sorting and entropycoding to yield the encoded base layer. Zigzag sorting prepares thequantized data in an efficient format for encoding. Entropy coding is acompression technique that uses a series of bit codes to represent a setof possible symbols. The enhanced layer result of transform/quantizationblock 1432 is also sent to an adder 1436, which subtracts the base layerand sends the result to a ZZ/entropy coder 1438 for the enhanced layer,as previously described in reference to FIGS. 31-36. Of further note,the enhanced layer is fed back (see line 1440 true rate update) forupdating the content classification 1420 of true rate and an operationfor determining long and short term histories of bit rates for used bythe rate control.

FIG. 17A is a flow diagram illustrating processing multimedia data thathas been obtained, received, or is otherwise accessible. Process 1700starts and at block 1702 it classifies content of the multimedia data.In one illustrative aspect, content classification can be performed byclassification means, for example, the content classification module 712in FIG. 7. Process 1700 continues on to block 1704 where it encodes themultimedia data in a first data group and a second data group based onthe content classification. This encoding is performed such that thefirst data group comprises a coefficient and the second data groupcomprises a first differential refinement associated with the first datagroup coefficient. This can be performed by encoding means describedherein, for example, the encoder 228 in FIG. 7.

FIG. 17B is a block diagram of a multimedia encoding system 1710 thatcan perform the process illustrated in FIG. 17A. In some aspects, themultimedia encoding system 1710 can be a transcoder, such as transcoder200. In other aspects, the encoding system 1710 can comprise a portionof a transcoder. The multimedia encoding system 1710 includes means forclassifying the content of multimedia data, module for classifyingcontent of multimedia data 1712. The means for classifying content canbe, for example, a classification module in a preprocessor (e.g.,preprocessor 226) or an encoder (e.g., encoder 228). The encoding system1710 also includes means for encoding the multimedia data, module forencoding the multimedia data 1714, which can be configured to encode themultimedia data in a first data group and a second data group based onthe content classification, where this encoding is performed such thatthe first data group comprises a coefficient and the second data groupcomprises a first differential refinement associated with the first datagroup coefficient. Other transcoder components, such as describedherein, also may be included in the encoding system 1710.

FIGS. 23, 24, 27 and 28 are process flow diagrams exemplifying methodsof encoding multimedia data that embody the aspects described herein.FIG. 23 is a process flow diagram illustrating a process 2300 ofencoding multimedia data based on the content information. At block 2305process 2300 receives encoded multimedia data, and at block 2310 process2300 decodes the multimedia data. At block 2315, process 2300 determinescontent information associated with the decoded multimedia data. Atblock 2320, process 2300 encodes the multimedia data based on thecontent information.

FIG. 24 is a process flow diagram illustrating a process 2400 ofencoding multimedia data so as to align data boundaries based on contentinformation level. At block 2405, process 2400 obtains contentinformation associated with the multimedia data, which can be done, forexample, by the preprocessor 226 or the content classification module712 shown in FIG. 7. At block 2410, process 2400 encodes the multimediadata so as to align data boundaries based on the content information.For example, slice boundaries and access unit boundaries are alignedwith frame boundaries based on a content classification of themultimedia data being encoded. The encoded data is then available forsubsequent processing and/or transmission to a mobile device, andprocess 2400 ends.

FIG. 27 is a process flow diagram illustrating a process 2700 forencoding data using an adaptive intra refresh scheme based on contentinformation. When process 2700 starts the multimedia data has beenobtained. At block 2705, process 2700 obtains content information of themultimedia data. Obtaining the content information can be performed by,for example, preprocessor 226 or content classification module 712 asdescribed above. Process 2700 proceeds to block 2710, where it encodesthe multimedia data using an adaptive intra refresh error resiliencescheme, where the adaptive intra refresh error resilience scheme isbased on the content information. The functionality of block 2710 can beperformed by the encoder 228. The encoded data is made available forsubsequent processing and transmission, and process 2700 then ends.

FIG. 28 is a process flow diagram illustrating a process of encodingmultimedia data using redundant I frames based on multimedia contentinformation. When process 2800 starts the multimedia data is availablefor processing. At block 2805, the process 2800 obtains contentinformation of the multimedia data. As described above, this can be doneby, for example, the preprocessor 226 and/or the encoder 228. At block2810, process 2800 encodes the multimedia data so as to insert one ormore additional I-frames into the encoded data based on the contentinformation. This can be done by the encoder 228 as described above inconnection with an error resiliency scheme, inserting the I-frames intothe base layer or the enhancement layer depending on the errorresiliency scheme employed. After block 2810, the encoded data isavailable for subsequent processing and/r transmission to a mobiledevice.

It should be noted that the methods described herein may be implementedon a variety of communication hardware, processors and systems known byone of ordinary skill in the art. For example, the general requirementfor the client to operate as described herein is that the client has adisplay to display content and information, a processor to control theoperation of the client and a memory for storing data and programsrelated to the operation of the client. In one aspect, the client is acellular phone. In another aspect, the client is a handheld computerhaving communications capabilities. In yet another aspect, the client isa personal computer having communications capabilities. In addition,hardware such as a GPS receiver may be incorporated in the client toimplement the various aspects. The various illustrative logics, logicalblocks, modules, and circuits described in connection with the aspectsdisclosed herein may be implemented or performed with a general purposeprocessor, a digital signal processor (DSP), an application specificintegrated circuit (ASIC), a field programmable gate array (FPGA) orother programmable logic device, discrete gate or transistor logic,discrete hardware components, or any combination thereof designed toperform the functions described herein. A general-purpose processor maybe a microprocessor, but, in the alternative, the processor may be anyconventional processor, controller, microcontroller, or state machine. Aprocessor may also be implemented as a combination of computing devices,e.g., a combination of a DSP and a microprocessor, a plurality ofmicroprocessors, one or more microprocessors in conjunction with a DSPcore, or any other such configuration.

The various illustrative logics, logical blocks, modules, and circuitsdescribed in connection with the aspects disclosed herein may beimplemented or performed with a general purpose processor, a digitalsignal processor (DSP), an application specific integrated circuit(ASIC), a field programmable gate array (FPGA) or other programmablelogic device, discrete gate or transistor logic, discrete hardwarecomponents, or any combination thereof designed to perform the functionsdescribed herein. A general-purpose processor may be a microprocessor,but, in the alternative, the processor may be any conventionalprocessor, controller, microcontroller, or state machine. A processormay also be implemented as a combination of computing devices, e.g., acombination of a DSP and a microprocessor, a plurality ofmicroprocessors, one or more microprocessors in conjunction with a DSPcore, or any other such configuration.

The disclosed methods and apparatus provide transcoding of video dataencoded in one format to video data encoded to another format where theencoding is based on the content of the video data and the encoding isresilient to error. The methods or algorithms described in connectionwith the examples disclosed herein may be embodied directly in hardware,in a software module executed by a processor, firmware, or in acombination of two or more of these. A software module may reside in RAMmemory, flash memory, ROM memory, EPROM memory, EEPROM memory,registers, a hard disk, a removable disk, a CD-ROM, or any other form ofstorage medium known in the art. An exemplary storage medium is coupledto the processor, such that the processor can read information from, andwrite information to, the storage medium. In the alternative, thestorage medium may be integral to the processor. The processor and thestorage medium may reside in an ASIC. The ASIC may reside in a userterminal. In the alternative, the processor and the storage medium mayreside as discrete components in a user terminal.

The examples described above are merely exemplary and those skilled inthe art may now make numerous uses of, and departures from, theabove-described examples without departing from the inventive conceptsdisclosed herein. Various modifications to these examples may be readilyapparent to those skilled in the art, and the generic principles definedherein may be applied to other examples, e.g., in an instant messagingservice or any general wireless data communication applications, withoutdeparting from the spirit or scope of the novel aspects describedherein. Thus, the scope of the disclosure is not intended to be limitedto the examples shown herein but is to be accorded the widest scopeconsistent with the principles and novel features disclosed herein. Theword “exemplary” is used exclusively herein to mean “serving as anexample, instance, or illustration.” Any example described herein as“exemplary” is not necessarily to be construed as preferred oradvantageous over other examples. Accordingly, the novel aspectsdescribed herein is to be defined solely by the scope of the followingclaims.

The invention claimed is:
 1. A method of encoding multimedia data,comprising: classifying content of multimedia data; partitioning themultimedia data into a first data group and a second data group based onthe content classification, wherein the partitioning is definedaccording to a bandwidth ratio; and encoding the multimedia data in thefirst data group and in the second data group based on the partitioning,wherein the amount of data in the first data group relative to thesecond data group corresponds to the bandwidth ratio.
 2. The method ofclaim 1, wherein the first data group comprises a coefficient and thesecond data group comprises a first differential refinement associatedwith the first data group coefficient.
 3. The method of claim 2, whereinthe first differential refinement indicates a difference between aselected video frame and frame data resulting from decoding the firstdata group.
 4. The method of claim 1, wherein the encoding comprisesdetermining a bit rate based on the content classification of themultimedia data, and encoding the multimedia data based on the bit rate.5. The method of claim 1, wherein classifying content comprisesdetermining complexity of the multimedia data.
 6. The method of claim 5,wherein the complexity comprises temporal complexity or spatialcomplexity.
 7. The method of claim 5, wherein the complexity comprisestemporal complexity and spatial complexity.
 8. The method of claim 1,wherein encoding comprises encoding the multimedia data so as to allowdecoding of only the first data group or decoding of the first datagroup and the second data group into a single combined data group. 9.The method of claim 1, wherein the first data group is a base layer andthe second data group is an enhancement layer.
 10. The method of claim9, further comprising: selecting the coefficient from one of an originalbase layer residual error coefficient or an original enhancement layerresidual error coefficient; and calculating the first differentialrefinement based on the coefficient and the original enhancement layerresidual error coefficient.
 11. The method of claim 1, wherein encodingfurther comprises encoding macroblock header information and motionvector information in the first data group.
 12. The method of claim 1,wherein encoding further comprises quantizing the first data group at afirst step size, and quantizing the second data group at a second stepsize, wherein the first step size and second step size are related by ascale factor.
 13. The method of claim 1, wherein encoding furthercomprises determining a first quantization parameter having a firstquantization step size for use in encoding the first data group, anddetermining a second quantization parameter having a second quantizationstep size for use in encoding the second data group, wherein the firstand second quantization parameters are determined based on contentinformation of a selected frame data, and wherein the first quantizationstep size is coarser than the second quantization step size.
 14. Themethod of claim 1, wherein encoding comprises encoding the first datagroup using I-frames, and P-frames or any combination thereof andencoding the second data group using I-frames, P-frames, and B-frames orany combination thereof.
 15. An apparatus for encoding multimedia data,comprising: means for classifying content of multimedia data; means forpartitioning the multimedia data into a first data group and a seconddata group based on the content classification, wherein the partitioningis defined according to a bandwidth ratio; means for encoding themultimedia data in the first data group and in the second data groupbased on the partitioning wherein the amount of data in the first datagroup relative to the second data group corresponds to the bandwidthratio, wherein at least one of the means is implemented at leastpartially in hardware of the apparatus, wherein at least some of themeans for classifying, the means for partitioning and the means forencoding is implemented in hardware.
 16. The apparatus of claim 15,wherein the first data group comprises a coefficient and the second datagroup comprises a first differential refinement associated with thefirst data group coefficient.
 17. The apparatus of claim 16, wherein thefirst differential refinement indicates a difference between a selectedvideo frame and frame data resulting from decoding the first data group.18. The apparatus of claim 16, wherein the first data group is a baselayer and the second data group is an enhancement layer.
 19. Theapparatus of claim 18, wherein the encoding means comprises: means forselecting the coefficient from one of an original base layer residualerror coefficient or an original enhancement layer residual errorcoefficient; and means for calculating the first differential refinementbased on the coefficient and the original enhancement layer residualerror coefficient.
 20. The apparatus of claim 15, wherein the encodingmeans comprise means for determining a bit rate based on the contentclassification of the multimedia data, and encoding the multimedia databased on the bit rate.
 21. The apparatus of claim 15, wherein theclassifying content means comprises means for determining complexity ofthe multimedia data, and wherein the selected multimedia data is encodedbased on the complexity of the multimedia data.
 22. The apparatus ofclaim 21, wherein the complexity comprises temporal complexity orspatial complexity.
 23. The apparatus of claim 21, wherein thecomplexity comprises temporal complexity and spatial complexity.
 24. Theapparatus of claim 15, wherein the encoding means comprises means forencoding the multimedia data so as to allow decoding of only the firstdata group or decoding of the first data group and the second data groupinto a single combined data group.
 25. The apparatus of claim 15,wherein the encoding means comprises means for encoding macroblockheader information and motion vector information in the first datagroup.
 26. The apparatus of claim 15, wherein the encoding means furthercomprises means for quantizing the first data group at a first stepsize, and quantizing the second data group at a second step size,wherein the first step size and second step size are related by a scalefactor.
 27. The apparatus of claim 15, wherein the encoding meanscomprises means for determining a first quantization parameter having afirst quantization step size for use in encoding the first data group,and determining a second quantization parameter having a secondquantization step size for use in encoding the second data group,wherein the first and second quantization parameters are determinedbased on content information of a selected frame data, and wherein thefirst quantization step size is coarser than the second quantizationstep size.
 28. The apparatus of claim 15, wherein the encoding meanscomprises: means for encoding the first data group using I-frames andP-frames; and means for encoding the second data group using I-frames,P-frames, and B-frames.
 29. An apparatus configured to encode multimediadata, comprising: a content classifying module configured to classifycontent of multimedia data and provide content classification data; apartitioner that partitions the multimedia data into a first data groupand a second data group based on the content classification, wherein thepartitioning is defined according to a bandwidth ratio; and an encoderconfigured to encode the multimedia data in the first data group and inthe second data group based on the partitioning, wherein the amount ofdata in the first data group relative to the second data groupcorresponds to the bandwidth ratio.
 30. The apparatus of claim 29,wherein the first data group comprises a coefficient and the second datagroup comprises a first differential refinement associated with thefirst data group coefficient.
 31. The apparatus of claim 30, wherein thefirst differential refinement indicates a difference between a selectedvideo frame and frame data resulting from decoding the first data group.32. The apparatus of claim 29, wherein the encoder comprises a bit ratecomponent configured to determine a bit allocation based on the contentclassification, and wherein the encoding component is further configuredto encode the selected multimedia data using the bit allocation.
 33. Theapparatus of claim 29, wherein classifying content comprises determiningcomplexity of the multimedia data, and wherein the selected multimediadata is encoded based on the complexity of the multimedia data.
 34. Theapparatus of claim 33, wherein the complexity comprises temporalcomplexity or spatial complexity.
 35. The apparatus of claim 33, whereinthe complexity comprises temporal complexity and spatial complexity. 36.The apparatus of claim 29, wherein encoding comprises encoding themultimedia data so as to allow decoding of only the first data group ordecoding of the first data group and the second data group into a singlecombined data group.
 37. The apparatus of claim 29, wherein the firstdata group is a base layer and the second data group is an enhancementlayer.
 38. A non-transitory machine-readable storage medium comprisinginstructions that, upon execution by a processor, cause the processorto: classify content of multimedia data; partition the multimedia datainto a first data group and a second data group based on the contentclassification, wherein the partitioning is defined according to abandwidth ratio; and encode the multimedia data in the first data groupand in the second data group based on the partitioning, wherein theamount of data in the first data group relative to the second data groupcorresponds to the bandwidth ratio.
 39. The non-transitorymachine-readable storage medium of claim 38, wherein the first datagroup comprises a coefficient and the second data group comprises afirst differential refinement associated with the first data groupcoefficient.
 40. The non-transitory machine-readable storage medium ofclaim 38, wherein the instructions to encode comprise instructions todetermine a bit allocation based on the content classification, andwherein the encoding component is further configured to encode theselected multimedia data using the bit allocation.
 41. Thenon-transitory machine-readable storage medium of claim 38, whereinclassifying content comprises determining complexity of the multimediadata, and wherein the selected multimedia data is encoded based on thecomplexity of the multimedia data.
 42. The non-transitory machinereadable storage medium of claim 41, wherein the complexity comprisestemporal complexity or spatial complexity.
 43. The non-transitorymachine-readable storage medium of claim 41, wherein the complexitycomprises temporal complexity and spatial complexity.
 44. A processorconfigured to: classify content of multimedia data; partition themultimedia data into a first data group and a second data group based onthe content classification, wherein the partitioning is definedaccording to a bandwidth ratio; and encode the multimedia data in thefirst data group and in the second data group based on the partitioning,wherein the amount of data in the first data group relative to thesecond data group corresponds to the bandwidth ratio.
 45. The processorof claim 44, wherein the first data group comprises a coefficient andthe second data group comprises a first differential refinementassociated with the first data group coefficient.
 46. The processor ofclaim 44, wherein the processor is further configured to determine a bitallocation based on the content classification, and wherein the encodingcomponent is further configured to encode the selected multimedia datausing the bit allocation.
 47. The processor of claim 44, wherein theprocessor is further configured to determine complexity of themultimedia data, and wherein the content classification is based on thecomplexity of the multimedia data.
 48. The processor of claim 47,wherein the complexity comprises temporal complexity or spatialcomplexity.
 49. The processor of claim 47, wherein the complexitycomprises temporal complexity and spatial complexity.